j@254: * AMuSE j@254: ** amuse-database-admin + CHARM db persistence j@254: - Generalize mtp-admin? j@254: - How general should amuse-database-tools be? j@254: - General import/export API? j@254: *** TODO Use database user accounts j@254: *** TODO Move prototype versioning code from CHARM into amuse-database-admin j@254: j@254: Provide the logic to update and retrieve successive versions of j@254: constituents. Includes providing a way of deleting (updating, marking j@254: something as deleted) and destroying. The latter should require j@254: adequate permissions and also test that no analyses depend on them. j@254: j@254: ***** TODO Define and implement versioning logic j@254: j@254: Revision control. There needs to be some logic to work out the union, j@254: intersection, relative complement between successive versions of j@254: constituents. Surely best to do this server side? Use temporary j@254: tables? j@254: j@254: ***** TODO Simplify current code j@254: j@254: Column names could be in a variable: implementation-id parent-id j@254: ext-properties int-properties owner version creation-timestamp j@254: deletion-timestamp. j@254: j@254: Some macros would also be helpful here, particularly if it would be j@254: possible to abstract out the table creation and versioning stuff. That j@254: way, all database tables created within AMuSE (in backends, or j@254: specialized applications like SIA) could be created with a macro that j@254: automatically generates the necessary revision control columns, j@254: triggers and functions. j@254: j@254: ***** TODO Decide on the long-term aim of amuse-database-admin j@254: j@254: More adventurously, the database admin stuff here could form the basis j@254: of a general mechanism for creating tables and getting data into and j@254: out of the database. Each implementation could be stored in tables j@254: generated by the template CHARM reversion controlled table, with j@254: backend-specific columns. SIA is not a backend as such, it can take j@254: any other backend and generate data from that. Anyway, the database j@254: interaction is very similar. j@254: j@254: This could tie up with better overall integration of CHARM j@254: constituents within each implementation. We could also ensure that all j@254: compositions added to the database, and all their events have unique j@254: ids! I still think we need an amuse_implementation table though, along j@254: with consistent constructor methods specialised on packages - j@254: otherwise, how else do we know to create a geerdes-constituent or an j@254: mtp-constituent. Or maybe these should be properties of constituents? j@254: But we'd still need the methods specialized on packages, or at least j@254: we'd need to look up consistently named functions within individual j@254: packages. j@254: j@254: How possible or beneficial would this be? Might simplify things in the j@254: long run and shouldn't have to compromise the flexibility of j@254: individual implementations. This would be more about general work flow j@254: and database interaction. I was going to try and do something along j@254: these lines with the midi-db backend. j@254: j@254: Or perhaps it's better to stick with specialized implementation j@254: specific database functions with some code duplication? j@254: j@254: *** TODO Check all backends use *amuse-database* j@254: j@254: Sort out the passing around of the database variable. Basically, there j@254: should only be one database connection accessible to every backend, as j@254: presumably it would be very difficult to maintain consistency across j@254: multiple databases (e.g. for the results of general analysis tools or j@254: the CHARM constituents mechanism). j@254: j@254: The database connection could be passed in as an optional argument j@254: (defaulting to *amuse-database*). But that would require changing the j@254: existing API (e.g. get-composition). The way I've done this in midi-db j@254: is to make sure all the database functions explicitly use j@254: *amuse-database* for any database call, rather than defaulting to j@254: clsql:*default-database*. This allows other databases to be used for j@254: testing purposes by shadowing *amuse-database* within the backend j@254: package. j@254: j@254: ** Geerdes j@254: *** TODO Sort out general weirdness in the data j@254: - Geerdes compositions timepoint = 0 instead of timepoint of first event. j@254: - g-id-file-id 1004 is in 6/4, which tactus-duration treats as compound time! j@254: - g-id-file-id 3822 is in 4/64! j@254: - g-id-file-id 8261 has a 63/32 time signature! j@254: j@254: *** TODO Import new files j@254: ** Constituents j@254: *** DONE Are the floating constituent abstractions sensible? j@254: - Or should the base constituent class not inherit from anchored period? j@254: j@254: *** TODO CHARM j@254: **** TODO Sort out old CHARM mess in my darcs Geerdes branch j@254: **** Questions j@254: j@254: 1. 'Parent' slot: the constituent the particles derived from j@254: (current implementation), or always the composition? j@254: j@254: 2. What to do about constituent properties? They are currently j@254: 'charm-property-list' objects, which are a subclass of j@254: sequence, but do not actually contain any additional slots! j@254: Should we define more structure for these objects? Should j@254: there be a differentiation between intrinsic and extrinsic j@254: lists? Should they be keyword-value lists instead of lists of j@254: strings? Or make better use of the class hierarchy? j@254: j@254: 3. Related to property-lists, how should we determine identity j@254: between constituents? Can this be done purely from the j@254: headers - in particular, the property lists, or is it j@254: necessary to also test for identity between all of the j@254: particles? j@254: j@254: 4. What to do about time? Should they be slots in the header? j@254: Time represented in database format (integer) or backend data j@254: type? j@254: j@254: 5. The need for a consistant API. If we are going to ask for j@254: specific constituents from the database, and these might be j@254: from any backend, there needs to be consistantly named j@254: functions like 'make-composition-identifier' (as opposed to j@254: g-id-file-id in geerdes) in each package. Examples of generic j@254: functions I've added are 'identifier' and 'id', which will j@254: return the identifier (implementation specific object) or j@254: database-id-number (integer) respectively for any j@254: 'amuse-database-object' given to them. (The class j@254: 'amuse-database-objects' currently does not exists, but it j@254: probably should, providing the slots that are shared between j@254: all AMuSE objects stored in a database.) j@254: j@254: Is this a bad thing to do? The semi-general API I work with is j@254: most similar to Geerdes, just because I was most familiar with j@254: that backend when I started. Marcus seems to have his own set j@254: of conventions, but the two are probably not incompatible. j@254: j@254: If we don't have a consistant API, we have to make a CHARM j@254: implementation within each backend? j@254: j@254: ** Future development j@254: *** Viewpoints, projections, properties and features j@254: - What is the difference between a viewpoint (MTP), a projection j@254: (SIA), and features (SIA, CONCEPTUAL-SPACES, and general j@254: constituent features and properties). j@254: - Retain original amuse context (like sia-datasets). j@254: - Define a general design for developing general tools?