Skip to content
This repository has been archived by the owner on Feb 26, 2022. It is now read-only.

cfx py

Jump to bottom
Gozala edited this page Jun 12, 2012 · 3 revisions

Brian Warner, @warner 03-Apr-2012

(note: this describes the state of Jetpack as of early April 2012, release 1.6 or 1.7, git revision [c37bf47de25551832b96a53d9cd15efe8bfa7c11] (https://github.com/mozilla/addon-sdk/tree/c37bf47de25551832b96a53d9cd15efe8bfa7c11). Things change fast in the SDK, and this document will probably be obsolete within a few months).

The Jetpack Addon-SDK contains a library of CommonJS modules (most of which provide access to specific browser APIs), and a "linker" which combines a loader module, user-written JS source code, and any referenced library modules. The linker outputs an .xpi file, ready for installation into a browser.

"cfx" is the main tool provided by the Addon-SDK. Users first interact with cfx by running "cfx init", which creates a skeleton addon source directory. After modifying the code, users run the "cfx xpi" subcommand in their source directory to run the linker and produce an .xpi file. When run as "cfx run", it produces an .xpi and immediately launches a browser with a new (generally ephemeral) profile, with the addon already installed. When run as "cfx test", it does the same, but executes the addon's unit test suite, then shuts down.

cfx has other modes that are less addon-specific. "cfx testall" runs the SDK's internal test suite, including tests on all library modules ("cfx testpkgs"), example code ("cfx testex"), and tests of cfx itself ("cfx testcfx"). "cfx docs" opens a browser window against a small built-in webserver, showing the rendered markdown documentation (both in docs/ and in packages/*/docs/). "cfx sdocs" renders this documentation into a static tarball full of HTML files, for copying to a normal webserver (like MDN).

cfx is implemented in Python, and the name is derived from "cuddlefish executable", where "cuddlefish" is the name of both the python package (SDK/python-lib/cuddlefish) that contains the bulk of the code, and the Javascript module (SDK/packages/api-utils/lib/cuddlefish.js) that implements most of the loader. "cuddlefish" derives from an amusing misspelling of "cuttlefish", an intriguing and potentially world-threatening species of fish.

The entry point is in bin/cfx, but that merely loads the right code and transfers control to SDK/python-lib/cuddlefish/__init__.py.

SDK python code layout

  • SDK/bin/cfx: entry point
  • SDK/python-lib/cuddlefish: most cfx code lives here
  • third-party modules:
  • SDK/python-lib/markdown: markdown->HTML renderer
  • SDK/python-lib/mozrunner: old snapshot of mozrunner, for 'cfx run'
  • SDK/python-lib/simplejson: JSON parser

"simplejson" is used in lieu of python's stdlib "json" module, because stdlib didn't include it until python2.6, and the SDK is supposed to be compatible with python2.5. Our "mozrunner" snapshot contains local modifications and has diverged a bit from the upstream (https://github.com/mozilla/mozbase).

python-lib/cuddlefish layout

boring things

  • tests/ contains unit tests, run by 'cfx testcfx'

  • app-extension/ contains the initial XPI contents, including a template for install.rdf and a copy of bootstrap.js, both of which serve as the XPI's entry point.

  • mobile-utils/ contains similar XPI contents for mobile addons.

  • docs/ contains parsers and renderers for the SDK's extended markdown format. These build on top of the third-party "markdown" library.

  • _version.py is used to automatically determine the SDK's version from git metadata. "cfx --version" reports this detailed version information.

  • bunch.py is a small utility class to make python dictionaries behave more like Javascript objects (mybunch.foo == mybunch["foo"]).

  • util.py holds some small functions to ignore boring files (dotfiles, editor tempfiles, version-control metadata) when scanning directories or building the XPI.

  • version_comparator.py is unused

  • templates.py holds the contents of the skeleton addon created by "cfx init"

  • options_defaults.py parses the "preferences" property of package.json

  • options_xul.py renders these parsed preferences into an options.xul file inside the XPI.

  • property_parser.py parses locale data from .properties files, for inclusion in XPI files like "locale/LANGUAGE.json".

  • prefs.py contains preferences which are installed into ephemeral "cfx run" profiles, mostly to turn on debugging tools and prevent the new browser from doing developer-hostile things like rejecting third-party-installed addons and immediately looking for updates.

  • rdf.py helps create the XPI's install.rdf by incorporating addon-specific data into the template from app-extension/ .

  • preflight.py makes randomly-generated "JID" addon identifiers. Long ago, JIDs were actually public keys, and the pre-flight code ensured that each addon had a corresponding private key in ~/.jetpack, generating a keypair if necessary. That was removed before 1.0 shipped.

big stuff

The bulk of cfx lives in 5 files:

  • __init__.py
  • manifest.py
  • packaging.py
  • runner.py
  • xpi.py

The first half of __init__.py contains options-parsers (using the stdlib "optparse" module) that populate the "options" object, and utility functions that drive the various unit-test modes. There is also a helper function to initialize a skeleton addon (invoked by "cfx init"). The other half of __init__.py is the large run() function.

SDK/bin/cfx transfers control to this run(). Most of the non-XPI-building subcommands (init, test, docs) are dispatched early. Once we're committed to building an XPI, the next step is to scan the top-level package.json and the contents of our packages/ directory.

We use the packaging.py module (packaging.get_config_in_dir()) to scan the top-level package.json file and create a data structure named "target_cfg". The actual return value is a "Bunch" (basically just a dictionary with more JS-like accessor properties). We'll call this kind of Bunch (returned by get_config_in_dir()) a "package descriptor". Note that this function only handles the main add-on source code (in the current directory), not the rest of packages/ (those are handled later). The function looks briefly in the package for some well-known directories ("tests" vs "test", etc), and for some icon files, but in general the only file it actually reads is package.json. It remembers the directory that the package lives in, as target_cfg.root_dir. Any keys in the package.json are copied to properties of target_cfg. We'll call

Back in __init__.py, we run some additional code that differentiates between the "xpi", "test", and "run" subcommands, specifically to figure out if we need to look for a "main" property or not. Then we delegate back to packaging.build_config() to find the rest of the packages.

build_config() performs a recursive scan for packages. It starts with SDK/packages/ and anything added to the --package-path. For each one, it finds all the subdirectories with package.json files and processes each subdirectory as a package. If the package contains a packages/ subdirectory, or names some other subdirectory via a .packages property in its package.json file, then that subdirectory will be scanned too. build_config() returns a pkg_cfg with the full set of known packages in pkg_cfg.packages (so a package named "foo" will be described by pkg_cfg.packages["foo"]). Each package descriptor is created by get_config_in_dir() as above. Note that name collisions between packages are not allowed: build_config() will throw an exception if this happens.

Back in __init__.py, the code now calls preflight_config() to make sure the JID and (what used to be) other data in package.json is ready. If the preflight checks fail, cfx prints a message (telling the user to fix the problem and then try again), then cfx is halted. This used to be necessary when the JID was a pubkey; in some situations the user needed to edit their package.json to change the key. Now that we no longer use keys, this code should be replaced (bug 613587) by having "cfx init" create a JID earlier, and the preflight check can merely assert that the addon has a legal JID.

Next, run() calls packaging.get_deps_for_target() to build a list of required packages for this addon. This uses package.json .dependencies and .extra_dependencies properties to traverse the package-dependency graph. It also includes SDK/packages/test-harness if the "test" subcommand is executing. The results are stored in "deps", as a list of package descriptors.

The next step is to build the manifest. run() first locates the cuddlefish.js loader module, and determines if "cfx test" is being run (to tell the manifest scanner to look for tests too). This it imports manifest.py and invokes the build_manifest() function, which returns a Manifest object. This process is examined in more detail below.

After generating the manifest, run() calls packaging.generate_build_for_target. This basically generates a list of directories ("sections") that need to be included in the XPI, in build.packages[pkgname][section]. It also builds a list of locale objects, and populates a few properties (icons, preferences) that need to be stored in harness-options.json.

Then run() creates the main harness_options object, filling in metadata pieces like the JID, the addon's name, entry point names, the sdkVersion, and the manifest itself. Then it calls rdf.gen_manifest to create the "RDF manifest" (i.e. install.rdf), which contains some of this metadata. If there is an update URL (for automatic updates of the addon), it stashes this into install.rdf too.

The next step depends upon the subcommand being run. For "cfx xpi", it asks the manifest for a list of used files, then calls xpi.build_xpi to create the zipfile and fill it with both synthesized metadata files and copies of files from user code and the packages/ library. For both "cfx run" and "cfx test", it delegates to runner.run_app to create a new profile (or reuse an existing one), build an XPI as with "cfx xpi", install the addon into the new profile, then launch firefox (or other application).

build_xpi() is a fairly straightforward recursive copy into a temporary directory, which is then bundled into a zipfile with a .xpi suffix. The current code walks through all packages and sections, doing a local os.walk() scan for files in its directory, copying everything that is listed in the manifest into the zipfile. It also copies locale files in. One proposed rewrite (by warner, not completed) changed this process to have the manifest create a "compile map" as a side-effect of its normal scan: a mapping from local-disk filename to XPI filename, then have xpi.py strictly follow this mapping to build the XPI. A reverse map named the "decompile map" would allow a hypothetical "cfx unpack" command to build a source tree out of the contents of the XPI, potentially making an XPI-repacker service more robust against changes to the XPI layout.

run_app() is built on top of mozrunner. It adds code to handle Fennec remote-debugger launching better, and to communicate with child browsers through logfiles (since apparently Gecko doesn't make it easy to write to stderr, and pipes on Windows are always troublesome). It is responsible for injecting the default preferences from prefs.py into the new profile. It includes a call to build_xpi, rather than using a single common build_xpi in __init__.py, because the filename of the logfile is passed to the addon via harness-options.json. If/when this is changed to pass through an environment variable instead, the local call should be replaced by a single common build_xpi() call.

"cfx test" is handled through the "cfx run" case, but the XPI generated (in particular the choice of entry point) will execute the test runner instead of the normal main() function.

Finally, when run_app() or build_xpi() returns, run() calls sys.exit(), and cfx terminates.

Building The Manifest

manifest.ManifestBuilder performs a recursive traversal of the module graph. It starts in the build() method, where it processes the entry points (the main.js module, any test files that weren't filtered away, and any extra loader modules). Each entry point can reference other modules which reference still more modules, etc. The process accumulates information about each module as instance attributes on the ManifestBuilder. Each module gets a ModuleInfo instance. The manifest itself (a list of (X,Y,Z) tuples that mean "when module X says require(Y), give it module Z") is stored as a numbered list of ManifestEntry instances. The code allocates a manifest entry before descending into the about-to-be-processed module, to tolerate cycles.

To process each module, ManifestBuilder.process_module() uses a regexp (scan_requirements_with_grep) to identify require() statements and extract the name of the module being imported in each. It then figures out which file will satisfy the import (ManifestBuilder.find_req_for()) by evaluating relative pathnames and searching the package list. Modules that already exist in the manifest are skipped (terminating cycles). New modules are processed, recursively.

find_req_for and related methods (_get_module_from_package, _get_entrypoint_from_package, _search_packages_for_module, _find_module_in_package) spend a lot of time and complexity implementing the require() lookup functions: this is the compile-time module search algorithm. Any changes to the way that require() should work will need to modify these functions, hopefully to simplify them.

Any time a require(self) statement is encountered, a DataMap object for the current package is created (or pulled from the cache). This holds a list of all data files, along with their filenames and hashes.

Later, when the XPI is built, several ManifestBuilder accessor methods are used to extract information about what to put into the zipfile. get_used_files provides a list of all the .js and data files used by the addon: this is used to filter unused files out of the XPI. get_harness_options_manifest is used to get the primary JSONable manifest structure for storage in harness-options.json.

Clone this wiki locally