DATASCOOTER
Support This Project
Custom Search

English Home
Russian Home
Quickstart
Performance
Guide
Download
Example
Contacts

Guide



Introduction:
This article about a software made for effortless and flexible working with data bases - not only as common Object Relational Mapping tool. It gives a truly freedom in operations with data.

Why Java?
Because it is smart, powerfull open-source language with many libraries and huge communty. So it grows very quickly, appears many new and blowmind conceptions and obtains many fans and further developers.

Why Eclipse?
Because it has modular structure and many features lets a developer to make powerful and robust applications with maximal independency of one parts of program to another.

Common structure

First datascooter wos developed for a thick clients, so it wos initially parted on a following parts:
    Client
Responsible for interaction with customer or user, must offer wide and useful interface for programmers with different level of knowledge and working capabilities. It is responsible for building objects which may be right interpreted on a server for a data manipulating actions and responsible for converting responces of the server to a form required of customers. As Client may be treaten a org.datascooter.impl.DataManager.
    Server
Responsible for processing requests and interacting with Data Bases , managing connections and transactions even distributed. It must be maximally simple for speed and robustness. As Server may be treaten a org.datascooter.impl.SnipManager.
    Data Transfer Agent
Responsible for transferring requests from customer and responces from server. Requests and responces may be as synchronous as asynchronous, so it must be ojects capable to be stored between moments of producing and consuming and serializable for network transfer. Data base independent and consume as little as possible memory for transferring data. May be cashed in runtime for avoiding double interpretations of a Data Operations. As Data Transfer Agent may be treaten a org.datascooter.impl.Snip.
    Data Operation
Responsible for collecting what wants customer and may be interpreted by Client into Data Transfer Agent and may be saved in data base for further decisions about approving, rejecting or repeated executing. May be cashed in runtime for repeated executing on the Client. As Data Operation may be treaten a org.datascooter.impl.DataSnip.
    Mapping Holder
Responsible for store all mappings and retrieve for all others agents a mapping of required entity. As Mapping Holder may be treaten a org.datascooter.bundle.EntityMapper.
    Mapping Unit
Responsible for store information about fields of entity and it mapping on a corresponding data base fields. And information about entity relations for selection of depended entities in wide meaning - not only binded by foreign keys directly and statically. May be created and changed dynamically in runtime. As Mapping Unit may be treaten a org.datascooter.bundle.EntityBundle.
    Mapping Provider
Responsible for obtaining information about mapping from Class fields, methods or annotations or XML sources. Must generate Mapping Units and register it in the Mapping Holder. As Mapping Provider works all descendants of org.datascooter.inface.IBundleProvider.

Start Work

The Start Point of any work is DataScooter - singleton responsible for holding all other components of the system. DataScooter has three providers :

    dataSourceProvider - responsible for handling and delivery of org.datascooter.inface.IDataSource objects. This objects save all information required for connection with data source - url,login,password and contextId - type of data base.

    connectorProvider - responsible for handling and delivery of org.datascooter.inface.IContextConnector objects. This objects possess information about details of working with concrete data base and paticipate in obtaining connections and it's releasing. This implementations depended on concrete JDBC driver- in spite of standarts - degree of compliance may be different. The standart is too big and developers of data bases and drivers see as the prime different things. So context connectors fills in this gap.

    builderProvider - responsible for handling and delivery of org.datascooter.inface.ISnipBuilder objects. This implementations depended on concrete JDBC driver and possess information about details of building a queries for concrete data base.
Implementations of IContextConnector and ISnipBuilder may be found in individual plugins (or libraries) named according a data base context with jar files of JDBC drivers. By example if you want to work with DB2 data base so plugin org.datascooter.db.db2 must be placed in plugins directory of you application.

    defaultManager - Implementation of org.datascooter.inface.IDataManager must be created before use of DaataScooter by calling one of methods start(...) or setDefault(...)
DataManager may be created automatically by start or by constructor with parameters -ISnipManager and ISnipBuilder . Implementation of ISnipManager may be as local as Proxy object for remote method invocation.


    EntityMapper - singleton responsible for handling of org.datascooter.bundle.EntityBundle objects. This objects objects possess information about mapping of entities to data base tables. You may to add or change EntityBundles in runtime. By default EntityMapper must receive bundles from a set of org.datascooter.inface.IBundleProvider objects. This objects may obtain mapping from different sources depends of implementation - from XML,data base,file,annotations and so on.



Background:


    Agreements
Mapped classes must meet common JavaBean requirements - no args constructor, private fields with getters and setters...
    Settings and requirements
In the root of plugin there is a file datascooter.properties responsible for initial customization of datascooter:

    statementCasheSize
Size of cashe of prepared statements - on the server side - depends of data base engine and general set of queries and it variety
    casheManagementTimeout
Timeout of starting cleaning of statement cashe - on the server side - depends of count of users and variety of queries
    statementTimeToLive
Time of live of prepared statements - on the server side - depends of count of users and variety of queries
    connectionPoolSize
Size of pool of database connections - on the server side - depends of data base engine
    waitConnectionTimeout
Timeout of waiting a connection if all available is busy - on the server - side depends of data base engine
    batchSize
Batch size by manupulation with many count of objects - on the server - side depends of data base engine and common amount of data
    clientSnipCashing
On/Off cashing of the DataSnip On the Client side - on the client side - avoids of repeated query buildings
    snipPolicy
Customize a snipPolicy: EXECUTE,EXECUTE_AND_LOG,ONLY_LOG - on the client side - lets log DML operations or even banning it may impact on performance
    defaultStringLength
Default scale of String fields if it is not defined in mapping
    defaultBlobLength
Default scale of Byte fields if it is not defined in mapping
    defaultClobLength
Default scale of Big String fields if it is not defined in mapping
    useDefaultProvider
Use default mapping provider - on the client side org.datascooter.bundle.DefaultMappingProvider allows to automatically map not yet mapped objects according it's structure and/or annotations - if true - EntityNotMappedException will never occurs and any object will be mapped and stored .
    tableValidationPolicy
Use table validation policy: NEVER, BY_START, BY_BUNDLE_CHANGE - on the client side - use NEVER if you want to create data base structure by oneself with SQL DDL script otherwise datascooter will attend about tables which required for storing mapped classes.
    allowFlush
Allows to fill up objects via interface IFlushable - on the client side - shifts of responsibility of restoring inner structure of objects from datascooter to objects implements org.datascooter.inface.IFlushable - after delivery an array of data from data base. It may be faster but need some additional work.
    logDDL
Allows to log DDL events
    Mapping via extension points
It is first developed and may be the most advanced kind of the mapping. Very smart and useful Eclipse extension points editor makes it process easy. Just try extension point org.datascooter.mapping...
    Mapping via annotations
It is the most popular now JPA annotations , but you code become too dirty visually and this mixt of two independed levels of application logic makes it depended of each other. The main JPA annotations are implemented but this process is not completed yet.
    Zero config
It is lazy way - when you have not time for mapping - just set useDefaultProvider=true in properties file and any fields equipped with public getters and setters defined in this class will be stored or selected from data base. Table name will be equal a simple class name with underline before case change in class name: SmallBox - SMALL_BOX. There are one own simple annotation - org.datascooter.annotations.AContainerAttribute - it may be used with this case. This annotation lets to save and select any emedded objects which getters marked with this annotation in spit of it's existed relational links in the data base.

Simple work with data:


    Simplest start
		
		DataScooter.start(true, pluginDir, "h2", "C:/temp/mm/", "sa", "");
		
In the first string we've started DataScooter with params:
boolean  checkTables - true if you not sure about right tables in data base and trust to datascooter take care about it.
String  pluginDir - path to directory where stored plugins with data base context pugins
String  contextId - what type of data base you want to use
String  url, String user, String password - connection properties
if in the datascooter.properties had written useDefaultProvider=true just call

		
		EntityMapper.getBundle(Trade.class);
		
And in runtime will be created mapping for a class Trade. And table for this class will be verified(created or changed) after creation the mapping.

    Save/Insert
		
		Trade trade = new Trade("trade", "car", 10L, new Date(), new Date(), user.getId());
		DataScooter.getDefault().save(trade);
		

    Select
		
		List list = DataScooter.getDefault().list(Trade.class);
		

    Update
		
		DataScooter.getDefault().update(trade, "goods", "Toyota");
		

    Delete
		
		DataScooter.getDefault().delete(trade);
		DataScooter.getDefault().delete(Trade.class.getName(), "goods", "Toyota");
		

    Count,Sum,Avg,Min,Max
		
		DataScooter.getDefault().count(Trade.class);
		


Advanced work with data:

All data operations in DataScooter must be described previously in objects org.datascooter.impl.DataSnip, so almost all facade methods of DataManager can receive this objects too. Then it will be translated into compact objects org.datascooter.impl.Snip. And then Snip will be sent to SnipManager for execution and then come back with result of execution. Like Space shuttle. DataSnip may be treaten as data operations.
    Fetch
It is unique feature of datascooter for selection some sets of different entities in one query to Server side:
		
		DataSnip dataSnip = DataSnip.select(Trade.class.getName());
		dataSnip.whereId(trade1.getId());
		dataSnip.fetch(Contract.class.getName());
		dataSnip.fetch(Offer.class.getName());
		dataSnip.fetch(NaturalUser.class.getName());
		Snip snip = DataScooter.getDefault().retrieve(dataSnip);
		
Here we create object DataSnip for selection Trade with given Id and lists of Contracts,Offers and Users linked or not linked with this Trade. In real world it may be useful for offline work with trade when dictionaries from central data base are not available. Object Snip may be stored in a local data base or implemented.
    Fetch by result
Another unique feature of datascooter for selection of one or more objects with all related data from others tables in one query to Server side:
		
		DataSnip dataSnip = DataSnip.select(Trade.class.getName());
		dataSnip.whereId(trade1.getId());
		dataSnip.fetchByResult(Contract.class.getName()).fetchByResult(Offer.class.getName()).fetchByResult(
			NaturalUser.class.getName());
		Snip snip = DataScooter.getDefault().retrieve(dataSnip);
		
Here we create object DataSnip for selection Trade with given Id and lists of Contracts,Offers and Users linked with this Trade. So if there are some relations between this objects and this Trade - they will be selected. Possible relations must be described in a mapping. It must be direct relation between tables - not cross relation. Otherwise you must to specify cross-objects between related objects.
    Pagination
If count of objects is too big - you may use pagination:
		
		DataSnip select = DataSnip.select(Offer.class.getName());
		select.setFromRow(1);
		select.setFetchSize(2);
		DataScooter.getDefault().list(select);
		
Here we want to select Offers from 1 and fetch size 2...
Or it may be like this:
		
		DataScooter.getDefault().list(Offer.class.getName(), "name", "List2", 2, 18);
		

    Save or Update
If we not sure about existence trade1 in data base - we may to use this method.
		
		DataScooter.getDefault().saveOrUpdate(trade1);
		

    Safe/Delete all
If you has a list of objects and want to save/delete it all by one call - try to use this:
		
		DataScooter.getDefault().saveAll(list);
		DataScooter.getDefault().deleteAll(list);
		
It would be faster and easier than write cycle.
    Group fetch
It is a selection of group different data sets from different tables in one query to Server side:
		
		DataSnip dataSnip = DataSnip.group();
		dataSnip.fetch(Contract.class.getName());
		dataSnip.fetch(Offer.class.getName());
		dataSnip.fetch(NaturalUser.class.getName());
		Snip snip = DataScooter.getDefault().retrieve(dataSnip);
		
Here we create object DataSnip for selection lists of Contracts,Offers and Users. In real world it may be useful for ofline work with dictionaries from central data base which may become unavailable. Object Snip may be stored in a local data base or implemented.
    Pipe
		
		DataScooter.getInstance().putSource(
		new DataSource("h2pipe", "jdbc:h2:file:~/testPipe/data;LOCK_TIMEOUT=20000;LOG=0", "sa", "", "h2", false));
		IDataManager manager = DataScooter.getInstance().getManager("h2pipe");
		manager.verifyTables();
		manager.executeAs(DataScooter.getDefault().retrieve(DataSnip.select(Trade.class.getName())), SnipType.INSERT);
		
Here we may to see one of useful examples - piping data from ona data base to another. As preparation we add new DataSource and build DataManager for it, then we verifying tables and in the last string piping all trades from default DataManager to new one.
    Data Set
Another unique feature of datascooter - possibility of working with named data sets. Named data set may be named as "class without constructor" - a mapping of this entity the same as any class but without class name. So it wll never be instantiated, but you may to work with it as an object - you can save,select or delete it as usual. It may be useful when structure of data unknown at development time or too volatile.
		
		String entity = "someObject";
		EntityBundle bundle = new EntityBundle(entity, null, "SomeTable", null, true, false);
		bundle.setId(new SimpleAttribute("id", "id", DBType.CHAR, new Integer(36), null, null, null, Boolean.FALSE));
		bundle.addSimpleAttribute(new SimpleAttribute("firstx", "firstx", DBType.STRING, new Integer(55), null, null, null, null));
		bundle.addSimpleAttribute(new SimpleAttribute("secondx", "secondx", DBType.INT, null, null, null, null, null));
		DataScooter.getInstance().addBundle(bundle, true);
		DataScooter.getDefault().delete(entity);
		int i = 5555;
		String string = "atom";
		DataScooter.getDefault().saveSet(entity, UUID.randomUUID().toString(), string, i);
		Snip snip = DataScooter.getDefault().retrieve(DataSnip.select(entity));
		
So we in runtime may to construct entity = "someObject" and start working with it.
    Logic attributes
It is yet another unique feature of datascooter - it allows to build complicated logical trees for defining membership of data row in a table in some datascooter entity. On the client side it may have result as changing implementation class depends on some fields:
		
		DataScooter.getDefault().list(ChildTrade.class.getName());
		
In this example ChildTrade stored in the same table as Trade but operations of selecting and counting gives us different values according defined in mapping logical attribute.

Logical attribute for Trade:

Capture

Logical attribute for ChildTrade:

Capture
In Logical attribute we may to point any fields with any logic not only one discriminator as in other ORM tools.


    Log all data manipulations
DataSnip is serializable and yet mapped so it may be saved in any data base for further verification or rollback as operations in this data base as dsistributed transactions.
		
		DataScooter.getDefault().setSnipPolicy(SnipPolicy.EXECUTE_AND_LOG, "my");
		DataSnip update = DataSnip.update(Trade.class.getName()).whereId(trade3.getId()).what("goods", "Mersedes");
		DataScooter.getDefault().update(update);
		DataScooter.getDefault().update(trade4, "goods", "Volvo");
		DataScooter.getDefault().setSnipPolicy(SnipPolicy.EXECUTE, "my");
		
When we set SnipPolicy.EXECUTE_AND_LOG - all operations will be logged as saving DataSnips after execution. And after it we can select DataSnip and to execute it again:
		update = DataScooter.getDefault().firstById(DataSnip.class, update.getId());
		DataScooter.getDefault().execute(update);
		

    Snip Query
Also DataSnip may be described in the mapping in org.datascoter.query extension point and be read and executed after start. Any complicated queries and possibilities described below may be predefined.
		
		DSQueryProvider provider = new DSQueryProvider(new EclipseExtensionReader("org.datascooter.query"));
		provider.explore();
		DataSnip query = provider.getItem("selectOffer");
		Snip snip = DataScooter.getDefault().retrieve(query);
		

Here we instantiated DSQueryProvider and explore queries from extension point and run query "selectOffer".

Capture


    Save or Update Snip
Let's select Trade with given Id and Contracts and Offers and Users. After it let's remove all this objects from data dase and check it absense. And save Snip... Now all of removed objects exists again.
		
		DataSnip dataSnip = DataSnip.select(Trade.class.getName());
		dataSnip.whereId(trade1.getId());
		dataSnip.fetch(Contract.class.getName());
		dataSnip.fetch(Offer.class.getName());
		dataSnip.fetch(NaturalUser.class.getName());
		Snip snip = DataScooter.getDefault().retrieve(dataSnip);
		DataScooter.getDefault().delete(Trade.class);
		DataScooter.getDefault().delete(NaturalUser.class);
		assertTrue(DataScooter.getDefault().list(NaturalUser.class).size() == 0);
		assertTrue(DataScooter.getDefault().list(Trade.class).size() == 0);
		DataScooter.getDefault().saveOrUpdate(snip);
		Object firstById = DataScooter.getDefault().first(Trade.class);
		assertTrue(firstById.equals(trade1));
		list2 = DataScooter.getDefault().list(NaturalUser.class);
		assertTrue("User count = " + list2.size(), list2.size() == 4);
		


    Patial select
Sometimes it is not desirable to select whole object but only some required fields. No problem:
 
		
		DataSnip select = null;
		DataSnip select1 = null;
		try {
			select = DataSnip.select(NaturalUser.class.getName()).whereId(user2.getId()).exclude("lastName");
			select1 = DataSnip.select(NaturalUser.class.getName()).whereId(user2.getId()).include("lastName");
		} catch (EntityNotMappedException e) {
			e.printStackTrace();
		}
		user2 = DataScooter.getDefault().first(select);
		assertTrue(user2.getLastName() == null);
		user2 = DataScooter.getDefault().first(select1);
		assertTrue(" FirstName = " + user2.getFirstName(), user2.getFirstName() == null);
		

    References
org.datascooter.bundle.Reference - it is wrapper for DataSnip that allows to use for return any other objects stored in data base - may be not linked with foreign keys with caller. It may be List,Set,Map or One object. Just query, just select something objects. It must be mapped and may be used in org.datascooter.utils.ReferenceInvoker for invoke from methods of object and for lazy fetching.

Capture


    Container Attribute
Mmmmm such tasty thing!!! org.datascooter.bundle.ContainerAttribute - it is tool for saving and invoking embedded objects... By example- in runtime you have object Location and it has method Address getAddress() and object Address wos created in runtime and you want to save Location and Address too...
You may to mark this getter with annotation @AContainerAttribute or by other means make a mapping of ContainerAttribute for this getter and by saving the Location this getter will be invoked and Address will be saved too.
By selection from data base of object Location the same object Address(by Id) may be inserted in Location via setter depends of PersistencePolicy - ALL, SELECT, INSERT, DELETE, UPDATE, NONE. Even if in data base there are not any references between this tables... Or invoking of this getter may be intercepted by AspectJ and then invoked ReferenceInvoker for lazy fetching. This mechanism let us to save in data base any references between objects even unmapped beforehand. You may to forget about cross tables and blowmind SQL queries. ContainerAttribute responsible for separately exists ojects inside of other objects.

    Entity parts
org.datascooter.bundle.EntityPart - it is class purposed for defining a composite entities:

Capture

By example in domain model we have class NaturalUser inherited from User, so we may to map it individually or may to define EntityPart. And by saving of NaturalUser from instance will be invoked getters inherited from User and this values will be stored in the table mapped for User with the same Id(it may be the same table as for NaturalUser or another). In case of selection this actions will be executed in a reverse order. So entities may be stored by parts even represented Java classes has not inheritance - you must just define appropriated methods on each of them and make EntityParts in a mapping. As User is mapped separately it may be used separately - without NaturalUser.
EntityParts in a parent object is not exists as separately objects but as group of properties

    Embedded Entities
You may use @AContainerAttribute in that way in a class Address :
		
		@AContainerAttribute(embedded = true)
		public PostBox getBox() {
			return box;
		}
		
So PostBox will be stored in the same table as Address and it will exists in runtime as separately object inside of instance of Address.

    Reference Invoker
As decribed before org.datascooter.utils.ReferenceInvoker - it is utility class for lazy loading and instantiation of embedded objects defined in References,ContainerAttributes and so on. It may be used in code as:
		
		@Transient
		public NaturalUser getUser() {
			try {
				return ReferenceInvoker.invoke(this, "getUser");
			} catch (Exception e) {
				e.printStackTrace();
			}
			return null;
		}
		
Or may be invoked via AspectJ or another Proxy or interceptor of calling of methods.

    Transactions
The simple variant of transaction - it is not complete JPA and JTA implementation it is just dress rehearsal - but it working:
		
		DataScooter.getDefault().beginTransaction("first");
		DataScooter.getDefault().update(trade1, "goods", "Volvo12");
		DataScooter.getDefault().rollback();
		trade1 = DataScooter.getDefault().firstById(Trade.class, trade1.getId());
		assertTrue("trade1.getGoods = " + trade1.getGoods(), trade1.getGoods().equals("Subaru"));
		DataScooter.getDefault().beginTransaction("second");
		DataScooter.getDefault().update(trade1, "goods", "Volvo12");
		DataScooter.getDefault().commit();
		trade1 = DataScooter.getDefault().firstById(Trade.class, trade1.getId());
		assertTrue("trade1.getGoods = " + trade1.getGoods(), trade1.getGoods().equals("Volvo12"));
		
It is a simple alone transaction by default all time in th object DataScooter there is marker of transaction - it is just null string. If you call DataScooter.getDefault().beginTransaction("first"); - it mean transaction wos named and further executions will belong to this transaction(each Snip will contain this name of transaction) until you start transaction with other name or call rollback or commit. This calls sends info to SnipManager and sets transaction to null.

    Distributed Transactions
There is special class for manage dictributed transaction:  org.datascooter.tansaction.Transaction
		
		IDataManager manager = DataScooter.getInstance().getManager("h2pipe");
		DataSnip dataSnip = DataScooter.getDefault().getBuilder().createInsertSnip(trade44);
		Transaction transaction = new Transaction("test");
		transaction.addTask(new Task(manager, dataSnip));
		transaction.addTask(new Task(DataScooter.getDefault(), dataSnip));
		transaction.runSync();
		trade44 = DataScooter.getDefault().firstById(Trade.class, trade44.getId());
		assertTrue(trade44 != null);
		trade44 = manager.firstById(Trade.class, trade44.getId());
		assertTrue(trade44 != null);
		
Here we create Transaction with two Tasks - each for own DataManager. Here implemented mechanism of two phase transaction: After successful execution of all Tasks all of them receives commit request and then , if this operation will successfully completed - commit.


YourKit is kindly supporting open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products: YourKit Java Profiler and YourKit .NET Profiler