Virtual table concept in madIS
==============================

Virtual table can be any piece of code that returns its results as list of lists. It will be used in madSQL in the FROM part of a query.

Example::

  select * from file('data.tsv');

In this case data are produced from external sources. Many madis virtual tables take a query as an input, execute it and returns its results transformed.
(See :ref:`Query parameter <unoteqparameter>`). In that case the vtable "call" could be inverted in the query. (see :ref:`SQL extensions <inversion>`).

Example::

  rowidvt select * from table1 limit 3;


Creating a virtual table function
----------------------------------------

Virtual table fuctions are the most advanced operator in madIS. Existing virtual table functions can be organized by increasing implementation complexity in three categories:

    - Single boolean output vt functions with query input: Virtual table functions that take an input, do sth and return 1 or 0. Example table fuctions are :mod:`~functions.vtable.exec`

    - Iterative output: Vts that take input and returns many data (iterator of list).
    Its call of the iterator retuns a list with each row data. In this category falls :mod:`~functions.vtable.file` operator, as well as :mod:`~functions.vtable.expand`, :mod:`~functions.vtable.rowidvt`, etc.

    - Advanced: Advanced vt's that need the index capability of virtual tables. In this category falls only :mod:`~functions.vtable.cache` functions. These functions must implement virtual tables from scratch as described in
    `APSW documentation <http://apidoc.apsw.googlecode.com/hg/vtable.html>`_.


All virtual table functions are created in XXXX folder. Every virtual table function must be single module. 
Detailed documentation of virtual table API could be found in `APSW documentation <http://apidoc.apsw.googlecode.com/hg/vtable.html>`_.
In madIS there are used mostly two specific types of Virtual table functions, that helper libraries have been implemented to simplify
their implementation.
So implementing a virtual table function, is feasible either by following the `APSW API <http://apidoc.apsw.googlecode.com/hg/vtable.html>`_
and adding the flags for madIS, either by using the helper libraries, as explained below.
To continue reading, have a look to the Virtual table function API in APSW and play with toggle function and a virtual table call.
Example::

    mterm> toggle tracing;
    tracing is now: True
    Query executed in 0 min. 0 sec 4 msec

    mterm> select * from (file 'data.csv');
    executetrace(u"create virtual table temp.vt_144176563 using file('data.csv','automatic_vtable:1')")
    executetrace(u'select * from (SELECT * FROM vt_144176563 );',None)
    ...
    executetrace(u'drop table temp.vt_144176563;')
    Query executed in 0 min. 0 sec 14 msec

    mterm> toggle vtdebug;
    executetrace(u"SELECT toggle('vtdebug');",None)
    vtdebug is now: True
    Query executed in 0 min. 0 sec 4 msec

    mterm> select * from (file 'data.csv');
    ....

Toggle is a row function that toggles the value of :ref:`madIS global settings<>`

Important flags for madIS VT functions
--------------------------------------

  registered=True # set this flag to register in madis the function, if set to false the module is not loaded.

  external_stream=True # Set this flag if no query parameter can be used.





Impementing single boolean output virtual table function
--------------------------------------------------------

The

    def function()
    from madis.vtout import SourceNtoOne

    class SourceNtoOne:
        def __init__(self,func,boolargs=None,nonstringargs=None,needsescape=None,notsplit=None,connectionhandler=False,retalways=False):


To create iterative VT's the madIS vtiterable could be used that hide some of the implementation details, or it can be build from scratch as in APSW, documentation. (http://apsw.googlecode.com/svn/publish/vtable.html) If vtiterable is not used, no standard parameter parsing will be available.

MUST BE SET TO USE VT in madis:
    One VT - one file in vtable folder
    registered=True ( as in all operators)
    external_stream=True (if the operator does not accept query parameter)

    Source function must be implemented!!



::

    #Cursor Iterator

    class TableCursor: # INTERFACE
        def __init__(self): # when called for first time, it must probably identify the schema, if it is not dynamic
            pass
        def __iter__(self):
            pass
        def next(self):
            pass
        def close(self):
            pass

::


    class TableVT: #INTERFACE       
        
        def __init__(self,envdict,largs,dictargs): #DO NOT DO ANYTHING HEAVY it, will be called before querying the table
            pass

        def getdescription(self):
            """
            must return a list of tuples (column name, columntype), if type not available leave it empty or None eg.(col,)
            example:
            [('column1','int'),('column2',),('column3','text')]
            """
            return schemalist

        def open(self):
            """
            After the first time it is called getdescription function must return table instance schema
            Must returns VT cursor
            """
            return TableCursor()
        
        def destroy(self):  # Called when Table instance is destroyed (on drop statement), OPTIONAL
            pass

.. function:: TableVT.__init__(self,envdict,largs,dictargs)

Parameters:

    :envdict:
        Dictionary with enviromental variables (correspond to APSW parameters of VTModule.Create)
        :tablename: name of the table that will be created.
        :db: instance of the connection to the database
        :dbname: database name
        :modulename: The string name under which the module was registered???????

    :largs: table function list arguments

    :dictargs: dictionary table function named arguments eg. type:int -> dictargs['type']='int'



def Source():
    return SourceVT(TableVT,boolargs,nonstringargs,needsescape,notsplit)

class SourceVT:
    def __init__(self,table,boolargs=None,nonstringargs=None,needsescape=None,staticschema=False,notsplit=None):

.. params:

Parameters

    :table: Class implementing TableVT interface
    :boolargs: List of parameter names that will be chnged to True/False if they are accordingly t/f
    :nonstringargs: Dictionary of parameter names that will be tranformed according to their value , eg. string input to dialect object {'dialect':{'line':line(),'tsv':tsv(),'csv':defaultcsv()}}
    :needsescape: List of parameter names that that include escape sequences eg. \n, \t etc.
    :staticschema: True if TableVT.open does not need to be executed before getdescription can be called
    :notsplit: List of parameter names that should not be splitted in named parameters eg. ['http'] for do not splitting http://server.com







