Database file handling.
Bases: sumolib.JSON.Container
Detailed dependency information.
Taken from sumo-scan and from the build database. Datastructure:
{ "modulename": { "versionname": { "depmodule" :
{
"depvers1": state,
"depvers2": state
}
}
}
}
add a single dependency with a state.
Bases: sumolib.JSON.Container
the buildtree database.
add build data from another Builddb to this one.
Note: this does NOT do a deep copy, it copies just references.
return a new Builddb that satisfies the given list of specs.
Note that this function treats versions like “R1-3” and “1-3” to be different.
generate a new buildtag.
A new buildtag in the form “STEM-nnn” is generated.
returns True if there are links to other_build_tag.
return linked build_tag if the module is linked or None.
return the modules of a build in form module spec strings.
This function returns a list of strings that ccan be parsed by sumolib.ModuleSpec.Spec.from_string().
return all modules of a build.
The returned structure is a dictionary mapping modulenames to versionnames.
return a set of tags of all builds that recursively depend on this.
raise exception if obj doesn’t look like a builddb.
This does not do a complete check of the datastructure, it only ensures that the JSON datastructure isn’t something completely different.
Bases: sumolib.Builds.DB
Implement a builddb with overlays.
Overlays are other build databases that are added to the local build database but cannot be modified and whose build specifications are not saved when the object is saved.
get or set the overlay mode.
Configuration file support.
Bases: object
store options in a JSON file.
load from all files in filenames list.
Database file handling.
Bases: sumolib.JSON.Container
the dependency database.
add an alias for modulename:versionname.
add dependency for a module:version.
test if a set of modules is complete.
This means that all dependencies are part of the given modules.
moduledict is a dictionary mapping modulename–>versionname.
do nothing if the module is found, raise KeyError otherwise.
If versionname is None, just check that the modulename is known.
Take all versions of old_modulename to create modulename.
makes a sets_dict complete with respect to dependencies.
A sets dict has this form:
convert modulespecs to a sets dict:
{ modulename1 : set(version1,version2),
modulename2 : set(version1,version2),
}
For each dependency that is missing, this program creates a new entry in the sets dict which contains all possible versions for the missing module.
Returns a set of modulenames of added dependencies.
delete dependency for a module:version if it exists.
returns True if dependencies are found for modulename:versionname.
returns True if given dependency is found.
return the alias or the original name for dep_modulename.
copy the module data from another Dependencies object.
This does a deepcopy of the data.
return an iterator on dependency modulenames of a module.
merge another Dependencies object to self.
return a dict {type:dict} for the module source.
return a tuple (type,dict) for the module source.
return the source url or tar-file or path for a module.
take items from the Dependencies object and create a new one.
List must be a list of tuples in the form (modulename,versionname).
This function copies modules whose versionname match exactly the given name, so “R1-3” and “1-3” are treated to be different.
Note that the new Dependencies object only contains references of the data. This DOES NOT do a deep copy.
take items from the Dependencies object and create a new one.
modulespecs must be a sumolib.ModuleSpec.Specs object.
Note that this function treats versions like “R1-3” and “1-3” to be different.
If no versions are defined for a module, take all versions.
When no moduleversions are found, rause a ValueError exception.
Note that the new Dependencies object only contains references of the data. This DOES NOT do a deep copy so you should NOT modify the result.
add a new version to the database by copying the old one.
do_replace: if True, replace the old version with the new one
search module names and source URLS for a regexp.
Returns a list of tuples (modulename, versionname).
set sourcespec of a module, creates if it does not yet exist.
returns True if the spec was changed, False if it wasn’t.
try to change sourcespec by providing a tag.
returns True if the spec was changed, False if it wasn’t.
create a dict of sets according to modulespecs.
modulespecs must be a sumolib.ModuleSpec.Specs object.
convert modulespecs to a sets dict:
{ modulename1 : set(version1,version2),
modulename2 : set(version1,version2),
}
sorts modules by dependencies.
Order that dependent modules come after the modules they depend on.
moduleversions: a list of pairs (modulename, versionname).
Bases: sumolib.JSON.Container
convert the old dependency database to the new format.
returns a BuildCache and a Dependency object.
JSON utilities.
Bases: object
an object that is a python structure.
This is a dict that contains other dicts or lists or strings or floats or integers.
create an object from a json file.
load from a cPickle file, don’t use lockfiles or anything.
Bases: exceptions.Exception
This is raised when we cannot get consistent JSON data.
Bases: exceptions.Exception
This is raised when the JSON data is invalid.
Dump a variable in JSON format.
Here is an example:
>>> var= {"key":[1,2,3], "key2":"val", "key3":{"A":1,"B":2}}
>>> dump(var)
{
"key": [
1,
2,
3
],
"key2": "val",
"key3": {
"A": 1,
"B": 2
}
}
Dump a variable to a file in JSON format.
This function uses a technique to write the file atomically. It assumes that we have a lock on the file so the temporary filename does not yet exist.
module specifications.
Bases: object
a class representing a single module specification.
raise ValueError exception if spec is not exact.
An exact module specification is a specification where for a module there is exactly one version given.
Test if a version matches another version.
create modulespec from a string.
Here are some examples:
>>> Spec.from_string("ALARM")
Spec('ALARM',None,None)
>>> Spec.from_string("ALARM:R3-2")
Spec('ALARM','R3-2','eq')
>>> Spec.from_string("ALARM:+R3-2")
Spec('ALARM','R3-2','ge')
>>> Spec.from_string("ALARM:-R3-2")
Spec('ALARM','R3-2','le')
Test if a version matches the spec.
Here are some examples: >>> m= Spec.from_string(“ALARM:R3-2”) >>> m.test(“R3-1”) False >>> m.test(“R3-2”) True >>> m.test(“R3-3”) False
>>> m= Spec.from_string("ALARM:-R3-2")
>>> m.test("R3-1")
True
>>> m.test("R3-2")
True
>>> m.test("R3-3")
False
>>> m= Spec.from_string("ALARM:+R3-2")
>>> m.test("R3-1")
False
>>> m.test("R3-2")
True
>>> m.test("R3-3")
True
Bases: object
A class representing a list of Spec objects.
raise ValueError exception if not all spec are exact.
An exact module specification is a specification where for each module there is one version given.
raise ValueError exception if a module is found more than once.
This ensures that the module specifications have only one specification for each modulename.
scan a list of module specification strings.
returns a new Specs object.
Note that if a modulename is used twice, the later definition overwrites the first one. However, the module retains it’s position in the internal list of modules.
Here are some examples:
>>> def p(s):
... for m in s:
... print m
>>> p(Specs.from_strings(["A:R2","B:-R3","C:+R1"], None))
Spec('A','R2','eq')
Spec('B','R3','le')
Spec('C','R1','ge')
>>> p(Specs.from_strings(["A:R2","B:-R3","A:R3"], None))
Spec('A','R3','eq')
Spec('B','R3','le')
>>> p(Specs.from_strings(["A:R2","B:-R3",":rm:A"], None))
Spec('B','R3','le')
>>> p(Specs.from_strings(["A:R2","B:-R3",":rm:A","A:R3"], None))
Spec('A','R3','eq')
Spec('B','R3','le')
Command line interface.
Bases: sumolib.cli.Container
class for arguments and commands.
Bases: exceptions.Exception
Error from command line parsing.
Bases: object
a single command specification.
Bases: object
Spec for a command line option.
Bases: object
hold command specfications.
add a single specification.
define the special completion options.
Bases: sumolib.cli.Container
class for options.
return all files for a pattern for command completion
Test if a string “looks” like an option.
Here are some examples:
>>> is_opt("")
False
>>> is_opt("x")
False
>>> is_opt("-")
False
>>> is_opt("-$")
False
>>> is_opt("-x")
True
>>> is_opt("-xx")
False
>>> is_opt("--x")
True
>>> is_opt("--$")
False
>>> is_opt("--")
False
>>> is_opt("--x-s")
True
new command argument processing.
argspec must be None or an CmdSpecs object.
Note: This function contains sys.exit statements !!
Here is some testcode:
>>> tspec= CmdSpecs()
>>> tspec.add("MOD", completion= lambda x,r: [x+"a"+"-mod",x+"b"+"-mod"])
>>> tspec.add("VER", completion= lambda x,r: [x+"1"+"-ver",x+"2"+"-ver"],
... optional= True)
>>> tspec.add("ARG", completion= lambda x,r: [x+"a"+"-arg",x+"b"+"-arg"],
... optional= True, array= True)
>>> import pprint
>>> def t(cli_args, completion_mode):
... r=process_args(cli_args, tspec, completion_mode, True)
... if r:
... pprint.pprint(r)
>>> t([], None)
error, mandatory argument MOD missing
>>> t([], "word")
a-mod
b-mod
>>> t(["a"], "word")
aa-mod
ab-mod
>>> t(["mod"], None)
Arguments({'MOD': 'mod', 'VER': None, 'ARG': []})
>>> t(["mod"], "word")
moda-mod
modb-mod
>>> t(["mod"], "new")
1-ver
2-ver
>>> t(["mod","12"], None)
Arguments({'MOD': 'mod', 'VER': '12', 'ARG': []})
>>> t(["mod","12"], "word")
121-ver
122-ver
>>> t(["mod","12"], "new")
a-arg
b-arg
>>> t(["mod","12", "a"], None)
Arguments({'MOD': 'mod', 'VER': '12', 'ARG': ['a']})
>>> t(["mod","12", "a", "b"], None)
Arguments({'MOD': 'mod', 'VER': '12', 'ARG': ['a', 'b']})
>>> t(["mod","12", "a", "b"], "word")
ba-arg
bb-arg
>>> t(["mod","12", "a", "b"], "new")
a-arg
b-arg
process a single command.
cmd_list: list of possible commands (cli_args[0])
returns a tuple (cmd, args)
parse incomplete options.
expects args[0] to be the program’s name.
>>> spcs= OptionSpecs()
>>> spcs.completion_options("list","--list","--listnew")
>>> spcs.add("--help -h")
>>> spcs.add("--flag -F")
>>> spcs.add("-x", arg_name="VALUE")
>>> spcs.add("-a", arg_name="ELM", array= True)
>>> spcs.add("-y", lambda x,r: [x+"a",x+"b"], "YOPT")
>>> def t(args):
... a= ["dummy"]
... a.extend(args)
... (options,rest)= process_opts(a, spcs, True)
... if options is not None:
... print options
... print "rest:", repr(rest)
>>> t([])
Options
a: None
flag: None
help: None
list: None
x: None
y: None
rest: []
>>> t(["-h"])
Options
a: None
flag: None
help: True
list: None
x: None
y: None
rest: []
>>> t(["--help"])
Options
a: None
flag: None
help: True
list: None
x: None
y: None
rest: []
>>> t(["-F"])
Options
a: None
flag: True
help: None
list: None
x: None
y: None
rest: []
>>> t(["--flag"])
Options
a: None
flag: True
help: None
list: None
x: None
y: None
rest: []
>>> t(["-x"])
error, argument 'VALUE' missing for option '-x'
>>> t(["-x", "ab"])
Options
a: None
flag: None
help: None
list: None
x: 'ab'
y: None
rest: []
>>> t(["-a", "a"])
Options
a: ['a']
flag: None
help: None
list: None
x: None
y: None
rest: []
>>> t(["-a", "a", "-h", "-a", "b"])
Options
a: ['a', 'b']
flag: None
help: True
list: None
x: None
y: None
rest: []
>>> t(["-a", "a", "-F", "-a", "b"])
Options
a: ['a', 'b']
flag: True
help: None
list: None
x: None
y: None
rest: []
>>> t(["-F", "--", "-x"])
Options
a: None
flag: True
help: None
list: None
x: None
y: None
rest: ['-x']
>>> t(["-a", "--", "-x"])
Options
a: ['-x']
flag: None
help: None
list: None
x: None
y: None
rest: []
>>> t(["-a", "--", "-x", "a"])
Options
a: ['-x']
flag: None
help: None
list: None
x: None
y: None
rest: ['a']
>>> t(["-a", "--", "b"])
Options
a: ['b']
flag: None
help: None
list: None
x: None
y: None
rest: []
>>> t(["-a", "ab", "--", "-h"])
Options
a: ['ab']
flag: None
help: True
list: None
x: None
y: None
rest: []
>>> t(["-y","a"])
Options
a: None
flag: None
help: None
list: None
x: None
y: 'a'
rest: []
>>> t(["-y","a","--list"])
aa
ab
>>> t(["-y","a","--listnew"])
Options
a: None
flag: None
help: None
list: 'new'
x: None
y: 'a'
rest: []
>>> t(["-y","a","b","--listnew"])
Options
a: None
flag: None
help: None
list: 'new'
x: None
y: 'a'
rest: ['b']
>>> t(["-y","a","b","c","--listnew"])
Options
a: None
flag: None
help: None
list: 'new'
x: None
y: 'a'
rest: ['b', 'c']
>>> t(["-y","a","b","--list"])
Options
a: None
flag: None
help: None
list: 'word'
x: None
y: 'a'
rest: ['b']
>>> t(["--","--list"])
--flag
--help
>>> t(["-","--list"])
-F
-a
--flag
-h
--help
-y
-x
>>> t(["-a","--list"])
-a
Special sumo command completion functions.
create a module cache in $HOME/.sumo.modulecache if needed.
returns the cache data.
callback dummy func.
create a module cache in $HOME/.sumo.modulecache if needed.
returns the cache data.
callback dummy func.
try to complete a dependency for a given module.
darcs support
Bases: object
represent a darcs repository.
spec must be a dictionary with “url” and “tag” (optional).
return a Repo object if a darcs repo was found.
This function returns <None> if no working repo was found.
If bool(hints[“write check”]) is True, return <None> if the repository directory is not writable.
For parameter “hints” see comment at __init__.
urlsupport
git support
Bases: object
represent a git repository.
spec must be a dictionary with “url” and “tag” (optional).
return a Repo object if a git repo was found.
This function returns <None> if no working repo was found.
If bool(hints[“write check”]) is True, return <None> if the repository directory is not writable.
For parameter “hints” see comment at __init__.
Lockfile support.
Bases: exceptions.Exception
No rights to create a lock.
This is raised when we can’t create a lock due to access rights on the directory
Bases: exceptions.Exception
This is raised when we can’t get a lock.
Bases: object
Implement a simple file locking mechanism.
do the file locking.
On linux, create a symbolic link, otherwise a directory. The symbolic link has some information on the user, host and process ID.
On other systems the created directory contains a file whose name has some information on the user, host and process ID.
Scan definitions in a makefile or a list of makefiles. This module actually calls “make” in order to examine the files and returns the values of all variables or makefile macros as a dictionary.
The main function in this module is “scan”. “scan” is called like this:
data= scan(filenames, verbose, dry_run)
Parameters are:
scan makefile-like definitions.
This takes a makefile name or a list of makefile names and returns a dictionary with all definitions made in these files. All definitions are resolved meaning that all variables that are used in the values of definitions are replaces with their values.
mercurial support
Bases: object
represent a mercurial repository.
spec must be a dictionary with “url” and “tag” (optional).
return a Repo object if a mercurial repo was found.
This function returns <None> if no working repo was found.
If bool(hints[“write check”]) is True, return <None> if the repository directory is not writable.
For parameter “hints” see comment at __init__.
patch file support.
path support
Repository support
Bases: object
Object for managing data in a repository.
Do pull before read, commit and push after write.
Bases: sumolib.JSON.Container
hold the source specification.
set source spec by copying information from another object.
This can also handle wildcards.
returns True if the spec was changed, False if it wasn’t.
change the source spec just by providing a tag.
returns True if the spec was changed, False if it wasn’t.
scan a source specification.
A sourcespec is a list of strings. Currently we support here: [“path”,PATH] or [“tar”,TARFILE,{PATCHFILES}] or [<repotype>,URL] or [<repotype>,URL,TAG,{PATCHFILES}]
where <repotype> may be one of the strings in sumolib.repos.known_repos
Here are some examples:
>>> SourceSpec.from_string_sourcespec(["path","ab"])
SourceSpec({'path': 'ab'})
>>> SourceSpec.from_string_sourcespec(["path"])
Traceback (most recent call last):
...
ValueError: invalid source spec 'path'
>>> SourceSpec.from_string_sourcespec(["path","a","b"])
Traceback (most recent call last):
...
ValueError: invalid source spec 'path a b'
>>> SourceSpec.from_string_sourcespec(["darcs","abc"])
SourceSpec({'darcs': {'url': 'abc'}})
>>> SourceSpec.from_string_sourcespec(["darcs","abc","R1-2"])
SourceSpec({'darcs': {'url': 'abc', 'tag': 'R1-2'}})
>>> SourceSpec.from_string_sourcespec(["darcs"])
Traceback (most recent call last):
...
ValueError: invalid source spec 'darcs'
>>> SourceSpec.from_string_sourcespec(["darcs","abc","R1-2","xy"])
SourceSpec({'darcs': {'url': 'abc', 'tag': 'R1-2', 'patches': ['xy']}})
>>> SourceSpec.from_string_sourcespec(["darcs","abc","R1-2","xy","z"])
SourceSpec({'darcs': {'url': 'abc', 'tag': 'R1-2', 'patches': ['xy', 'z']}})
check out a working copy.
sourcespec must be a SourceSpec object.
scan a directory and return a repository object.
Hints must be a dictionary. This gives hints how the directory should be scanned. Currently we know these keys in the dictionary:
scan a directory and return a repository object.
Hints must be a dictionary. This gives hints how the directory should be scanned. Currently we know these keys in the dictionary:
System utilities.
execute a command with returncode.
execute a command and return the programs output may raise: IOError(errcode,stderr) OSError(errno,strerr) ValueError
tar file support
Utilities for the SUMO scripts.
Bases: object
apply one or more regexes on strings.
Bases: object
apply one or more regexes on strings.
ask if the user wants to abort the program.
Aborts the program if the user enters “y”.
ask a yes or no question.
return the current dir and change to a new dir.
If newdir is empty, return <None>.
add a key, create a set if needed.
Here is an example: >>> import pprint >>> d= {} >>> dict_of_sets_add(d,”a”,1) >>> dict_of_sets_add(d,”a”,2) >>> dict_of_sets_add(d,”b”,1) >>> pprint.pprint(d) {‘a’: set([1, 2]), ‘b’: set([1])}
change values from sets to sorted lists.
Here is an example: >>> import pprint >>> d= {‘a’: set([1, 2]), ‘b’: set([1])} >>> ld= dict_sets_to_lists(d) >>> pprint.pprint(ld) {‘a’: [1, 2], ‘b’: [1]}
update mydict with other but do not change existing values.
If keylist is given, update only these keys.
walk directories, follow symbolic links.
Implemented to behave like os.walk On Python newer than 2.5 os.walk can follow symbolic links itself.
checks if path is complient to Bessy convention for support paths.
Here are some examples: >>> is_standardpath(“support/mcan/2-3”, “R2-3”) True >>> is_standardpath(“support/mcan/2-3”, “R2-4”) False >>> is_standardpath(“support/mcan/head”, “R2-3”) False >>> is_standardpath(“support/mcan/2-3+001”, “R2-3”) True
update a list with another.
In the returned list each element is unique and it is sorted.
join command line option values to a single list.
Here is an example: >>> a=[“a b”,”c”,”d e f”] >>> opt_join(a) [‘a’, ‘b’, ‘c’, ‘d’, ‘e’, ‘f’]
convert a revision number to a comparable string.
This is needed to compare revision tags.
Examples of valid revisions:
2.3.4 2-3-4 R2-3-4 seq-2.3.4 head trunk
Here are some examples: >>> rev2key(“R2-3-4”) ‘002.003.004’ >>> rev2key(“2-3-4”) ‘002.003.004’ >>> rev2key(“head”) ‘-head’ >>> rev2key(“test”) ‘-test’ >>> rev2key(“test”)<rev2key(“R2-3-4”) True >>> rev2key(“R2-3-3”)<rev2key(“R2-3-4”) True >>> rev2key(“R2-3-5”)<rev2key(“R2-3-4”) False >>> rev2key(“R2-4-3”)<rev2key(“R2-3-4”) False >>> rev2key(“R1-3-4”)<rev2key(“R2-3-4”) True >>> rev2key(“R3-3-4”)<rev2key(“R2-3-4”) False
dict must have exactly one key, return it.
Raises ValueError if the dict has more than one key.
dict must have exactly one key, return it and it’s value.
Raises ValueError if the dict has more than one key.
split a path into (base, version, treetag).
Here are some examples: >>> split_path(“abc/def/1-3+001”) [‘abc/def’, ‘1-3’, ‘001’] >>> split_path(“abc/def/1-3”) [‘abc/def’, ‘1-3’, ‘’] >>> split_path(“abc/def/head+002”) [‘abc/def’, ‘head’, ‘002’]
split a path into head,treetag.
A path like /opt/Epics/R3.14.8/support/BIIcsem/1-0+001 is splitted to “/opt/Epics/R3.14.8/support/BIIcsem/1-0”,”001”
Here is an example: >>> split_treetag(“abc/def/1-0”) [‘abc/def/1-0’, ‘’] >>> split_treetag(“abc/def/1-0+001”) [‘abc/def/1-0’, ‘001’]