KaliVeda
1.13/01
Heavy-Ion Analysis Toolkit
|
Manage a collection of data files on a remote machine.
Data repositories are defined in the user's $HOME/.kvrootrc; some typical examples are given in $KVROOT/KVFiles/.kvrootrc:
DataRepository: ccali ccali.DataRepository.Type: remote ccali.DataRepository.RootDir: cchpssindra:/hpss/in2p3.fr/group/indra ccali.DataRepository.ReadProtocol: root ccali.DataRepository.XRDServer: ccxrdsn012:1999 ccali.DataRepository.XRDRootDir: /hpss/in2p3.fr/group/indra ccali.DataRepository.XRDTunnel.host: ccali.in2p3.fr ccali.DataRepository.XRDTunnel.port: 10000 ccali.DataRepository.XRDTunnel.user: ccali.DataRepository.RemoteAvailableRuns.protocol: curl ccali.DataRepository.RemoteAvailableRuns.url: http://indra.in2p3.fr/KaliVedaDoc ccali.DataRepository.FileTransfer.type: bbftp ccali.DataRepository.FileTransfer.server: ccbbftp.in2p3.fr ccali.DataRepository.FileTransfer.user:
<p<blockquote>
This is the definition of a "remote" data repository. A remote data repository is principally characterised by the fact that access to the data files is via some non-local protocol: in this case it is xrootd ("...ReadProtocol: root"). The host name and port of the xrootd server are given ("...XRDServer: ccxrdsn012:1999"), as well as the root directory to be used ("...XRDRootDir: /hpss/in2p3.fr/group/indra").
In order to use a remote repository, some way to access the database of available data files for each run type must be defined. Here it is the curl programme which is used to read them via a website ("...RemoteAvailableRuns.protocol: curl"; "...RemoteAvailableRuns.url: http://indra.in2p3.fr/KaliVedaDoc").
Another particularity of remote data repositories is that some additional stuff may be needed in order to be able to access the data. This is handled by KVRemoteDataRepository::IsConnected(). In the present example, an SSH tunnel is set up in order to allow secure connection to the xrootd server ("...XRDTunnel.*:").
Finally, transfer of data between data repositories is handled by KVDataTransfer and child classes. In the present example, this is configured to use bbftp ("...FileTransfer.type: bbftp"), for which the required details are given.
Definition at line 67 of file KVRemoteDataRepository.h.
Public Member Functions | |
KVRemoteDataRepository () | |
Default constructor. More... | |
virtual | ~ KVRemoteDataRepository () |
virtual Bool_t | CheckFileStatus (const KVDataSet *ds, const Char_t *datatype, const Char_t *runfile) |
virtual Bool_t | CheckSubdirExists (const Char_t *dir, const Char_t *subdir=0) |
virtual void | CopyFileFromRepository (const KVDataSet *ds, const Char_t *datatype, const Char_t *filename, const Char_t *destination) |
virtual int | CopyFileToRepository (const Char_t *source, const KVDataSet *ds, const Char_t *datatype, const Char_t *filename) |
virtual void | DeleteFile (const KVDataSet *ds, const Char_t *datatype, const Char_t *filename, Bool_t confirm=kTRUE) |
virtual KVUniqueNameList * | GetDirectoryListing (const KVDataSet *ds, const Char_t *datatype="", const Char_t *subdir="") |
virtual Bool_t | GetFileInfo (const KVDataSet *ds, const Char_t *datatype, const Char_t *runfile, FileStat_t &fs) |
virtual const Char_t * | GetFullPathToOpenFile (const KVDataSet *dataset, const Char_t *datatype, const Char_t *runfile) |
virtual const Char_t * | GetFullPathToTransferFile (const KVDataSet *dataset, const Char_t *datatype, const Char_t *runfile) |
virtual Bool_t | IsConnected () |
virtual Bool_t | IsRemote () const |
Returns kTRUE for remote repositories, kFALSE for local repositories. More... | |
virtual void | MakeSubdirectory (const KVDataSet *ds, const Char_t *datatype="") |
Public Member Functions inherited from KVDataRepository | |
KVDataRepository () | |
Default constructor. More... | |
virtual | ~ KVDataRepository () |
virtual Bool_t | CanWrite () const |
void | cd () |
virtual void | CommitFile (TFile *file, const Char_t *datatype, const KVDataSet *dataset) |
virtual int | CopyFile (const char *f, const char *t, Bool_t overwrite=kFALSE) |
void | CreateAllNeededSubdirectories (const KVDataSet *DataSet, const Char_t *DataType) |
virtual TFile * | CreateNewFile (const KVDataSet *dataset, const Char_t *datatype, const Char_t *filename) |
TSystem * | FindHelper (const char *path, void *dirptr=0) |
virtual KVDataSetManager * | GetDataSetManager () const |
Return pointer to data set manager for this repository. More... | |
virtual const Char_t * | GetFileTransferExec () const |
returns full path to executable used for remote file transfer More... | |
virtual const Char_t * | GetFileTransferServer () const |
returns server url used for remote file transfer More... | |
virtual const Char_t * | GetFileTransferType () const |
returns protocol used for remote file transfer More... | |
virtual const Char_t * | GetFileTransferUser () const |
returns user name used for remote file transfer More... | |
virtual const Char_t * | GetRootDirectory () const |
returns root directory of data repository (fLocalrootdir) More... | |
Bool_t | HelperIsConsistentWith (TSystem *helper, const char *path, void *dirptr=0) |
virtual Bool_t | Init () |
virtual KVAvailableRunsFile * | NewAvailableRunsFile (const Char_t *, const KVDataSet *) |
virtual TObject * | OpenDataSetRunFile (const KVDataSet *ds, const Char_t *type, Int_t run, Option_t *opt="") |
virtual void | Print (Option_t *opt="") const |
Print info on repository. More... | |
virtual void | PrintAvailableDatasetsUpdateWarning () const |
Public Member Functions inherited from KVBase | |
KVBase () | |
Default constructor. More... | |
KVBase (const Char_t *name, const Char_t *title="") | |
Ctor for object with given name and type. More... | |
KVBase (const KVBase &) | |
copy ctor More... | |
virtual | ~ KVBase () |
virtual void | Clear (Option_t *opt="") |
Clear object properties : name, type/title, number, label. More... | |
virtual void | Copy (TObject &) const |
Make a copy of this object. More... | |
const Char_t * | GetLabel () const |
UInt_t | GetNumber () const |
UInt_t | GetNumberOfObjects () const |
virtual TObject * | GetObject () const |
virtual const Char_t * | GetType () const |
Bool_t | HasLabel () const |
virtual Bool_t | IsCalled (const Char_t *name) const |
Bool_t | IsLabelled (const Char_t *l) const |
virtual Bool_t | IsType (const Char_t *typ) const |
virtual void | List () |
KVBase & | operator= (const KVBase &) |
copy assignment operator More... | |
Double_t | ProtectedGetX (const TF1 *func, Double_t val, int &status, Double_t xmin=0.0, Double_t xmax=0.0) const |
void | SetLabel (const Char_t *lab) |
virtual void | SetNumber (UInt_t num) |
virtual void | SetType (const Char_t *str) |
Public Member Functions inherited from TNamed | |
TNamed () | |
TNamed (const char *name, const char *title) | |
TNamed (const TNamed &named) | |
TNamed (const TString &name, const TString &title) | |
virtual | ~TNamed () |
virtual TObject * | Clone (const char *newname="") const |
virtual Int_t | Compare (const TObject *obj) const |
virtual void | FillBuffer (char *&buffer) |
virtual const char * | GetName () const |
virtual const char * | GetTitle () const |
virtual ULong_t | Hash () const |
virtual Bool_t | IsSortable () const |
virtual void | ls (Option_t *option="") const |
TNamed & | operator= (const TNamed &rhs) |
virtual void | SetName (const char *name) |
virtual void | SetNameTitle (const char *name, const char *title) |
virtual void | SetTitle (const char *title="") |
virtual Int_t | Sizeof () const |
Public Member Functions inherited from TObject | |
TObject () | |
TObject (const TObject &object) | |
virtual | ~TObject () |
void | AbstractMethod (const char *method) const |
virtual void | AppendPad (Option_t *option="") |
virtual void | Browse (TBrowser *b) |
ULong_t | CheckedHash () |
virtual const char * | ClassName () const |
virtual void | Delete (Option_t *option="") |
virtual Int_t | DistancetoPrimitive (Int_t px, Int_t py) |
virtual void | Draw (Option_t *option="") |
virtual void | DrawClass () const |
virtual TObject * | DrawClone (Option_t *option="") const |
virtual void | Dump () const |
virtual void | Error (const char *method, const char *msgfmt,...) const |
virtual void | Execute (const char *method, const char *params, Int_t *error=0) |
virtual void | Execute (TMethod *method, TObjArray *params, Int_t *error=0) |
virtual void | ExecuteEvent (Int_t event, Int_t px, Int_t py) |
virtual void | Fatal (const char *method, const char *msgfmt,...) const |
virtual TObject * | FindObject (const char *name) const |
virtual TObject * | FindObject (const TObject *obj) const |
virtual Option_t * | GetDrawOption () const |
virtual const char * | GetIconName () const |
virtual char * | GetObjectInfo (Int_t px, Int_t py) const |
virtual Option_t * | GetOption () const |
virtual UInt_t | GetUniqueID () const |
virtual Bool_t | HandleTimer (TTimer *timer) |
Bool_t | HasInconsistentHash () const |
virtual void | Info (const char *method, const char *msgfmt,...) const |
virtual Bool_t | InheritsFrom (const char *classname) const |
virtual Bool_t | InheritsFrom (const TClass *cl) const |
virtual void | Inspect () const |
void | InvertBit (UInt_t f) |
virtual Bool_t | IsEqual (const TObject *obj) const |
virtual Bool_t | IsFolder () const |
R__ALWAYS_INLINE Bool_t | IsOnHeap () const |
R__ALWAYS_INLINE Bool_t | IsZombie () const |
void | MayNotUse (const char *method) const |
virtual Bool_t | Notify () |
void | Obsolete (const char *method, const char *asOfVers, const char *removedFromVers) const |
void | operator delete (void *ptr) |
void | operator delete[] (void *ptr) |
void * | operator new (size_t sz) |
void * | operator new (size_t sz, void *vp) |
void * | operator new[] (size_t sz) |
void * | operator new[] (size_t sz, void *vp) |
TObject & | operator= (const TObject &rhs) |
virtual void | Paint (Option_t *option="") |
virtual void | Pop () |
virtual Int_t | Read (const char *name) |
virtual void | RecursiveRemove (TObject *obj) |
void | ResetBit (UInt_t f) |
virtual void | SaveAs (const char *filename="", Option_t *option="") const |
virtual void | SavePrimitive (std::ostream &out, Option_t *option="") |
void | SetBit (UInt_t f) |
void | SetBit (UInt_t f, Bool_t set) |
virtual void | SetDrawOption (Option_t *option="") |
virtual void | SetUniqueID (UInt_t uid) |
virtual void | SysError (const char *method, const char *msgfmt,...) const |
R__ALWAYS_INLINE Bool_t | TestBit (UInt_t f) const |
Int_t | TestBits (UInt_t f) const |
virtual void | UseCurrentStyle () |
virtual void | Warning (const char *method, const char *msgfmt,...) const |
virtual Int_t | Write (const char *name=0, Int_t option=0, Int_t bufsize=0) |
virtual Int_t | Write (const char *name=0, Int_t option=0, Int_t bufsize=0) const |
Protected Member Functions | |
virtual KVDataSetManager * | NewDataSetManager () |
Create and return pointer to new data set manager. More... | |
Protected Member Functions inherited from KVDataRepository | |
virtual int | Chmod (const char *file, UInt_t mode) |
virtual const Char_t * | GetReadProtocol (const Char_t *dataset, const Char_t *datatype) |
TObject * | OpenDataSetFile (const KVDataSet *ds, const Char_t *type, const TString &fname, Option_t *opt="") |
virtual void | PrepareXRDTunnel () |
void | SetFullPath (TString &path, const Char_t *protocol) |
Protected Member Functions inherited from TObject | |
virtual void | DoError (int level, const char *location, const char *fmt, va_list va) const |
void | MakeZombie () |
Additional Inherited Members | |
Public Types inherited from KVBase | |
enum | EKaliVedaBits { kIsKaliVedaObject = BIT(23) } |
Public Types inherited from TObject | |
enum | EDeprecatedStatusBits |
enum | EStatusBits |
Static Public Member Functions inherited from KVDataRepository | |
static KVDataRepository * | NewRepository (const Char_t *type) |
Static Public Member Functions inherited from KVBase | |
static Bool_t | AreEqual (Double_t x, Double_t y, Long64_t maxdif=1) |
Comparison between two 64-bit floating-point values. More... | |
static void | BackupFileWithDate (const Char_t *path) |
static void | CombineFiles (const Char_t *file1, const Char_t *file2, const Char_t *newfilename, Bool_t keep=kTRUE) |
static void | Deprecated (const char *method, const char *advice) |
static Bool_t | FindClassSourceFiles (const Char_t *class_name, KVString &imp_file, KVString &dec_file, const Char_t *dir_name=".") |
static Bool_t | FindExecutable (TString &exec, const Char_t *path="$(PATH)") |
static const Char_t * | FindFile (const Char_t *search, TString &wfil) |
static const Char_t * | GetBINDIRFilePath (const Char_t *namefile="") |
static const Char_t * | GetDATABASEFilePath () |
static const Char_t * | GetDATADIRFilePath (const Char_t *namefile="") |
static Bool_t | GetDataSetEnv (const Char_t *dataset, const Char_t *type, Bool_t defval) |
static const Char_t * | GetDataSetEnv (const Char_t *dataset, const Char_t *type, const Char_t *defval) |
static Double_t | GetDataSetEnv (const Char_t *dataset, const Char_t *type, Double_t defval) |
static const Char_t * | GetETCDIRFilePath (const Char_t *namefile="") |
static const Char_t * | GetExampleFilePath (const Char_t *library, const Char_t *namefile) |
Return full path to example file for given library (="KVMultiDet", "BackTrack", etc.) More... | |
static const Char_t * | GetINCDIRFilePath (const Char_t *namefile="") |
static const Char_t * | GetKVBuildDate () |
Returns KaliVeda build date. More... | |
static const Char_t * | GetKVBuildDir () |
Returns top-level directory used for build. More... | |
static const Char_t * | GetKVBuildTime () |
Returns KaliVeda build time. More... | |
static const Char_t * | GetKVBuildType () |
Returns KaliVeda build type (cmake build: Release, Debug, RelWithDebInfo, ...) More... | |
static const Char_t * | GetKVBuildUser () |
Returns username of person who performed build. More... | |
static const Char_t * | GetKVSourceDir () |
Returns top-level directory of source tree used for build. More... | |
static const Char_t * | GetKVVersion () |
Returns KaliVeda version string. More... | |
static const Char_t * | GetLIBDIRFilePath (const Char_t *namefile="") |
static const Char_t * | GetListOfPlugins (const Char_t *base) |
static const Char_t * | GetListOfPluginURIs (const Char_t *base) |
static const Char_t * | GetPluginURI (const Char_t *base, const Char_t *plugin) |
static void | GetTempFileName (TString &base) |
static const Char_t * | GetTEMPLATEDIRFilePath (const Char_t *namefile="") |
static const Char_t * | GetWORKDIRFilePath (const Char_t *namefile="") |
static const Char_t * | gitBranch () |
Returns git branch of sources. More... | |
static const Char_t * | gitCommit () |
Returns last git commit of sources. More... | |
static void | InitEnvironment () |
static bool | is_gnuinstall () |
static Bool_t | IsThisAPlugin (const TString &uri, TString &base) |
static TPluginHandler * | LoadPlugin (const Char_t *base, const Char_t *uri="0") |
static Bool_t | OpenContextMenu (const char *method, TObject *obj, const char *alt_method_name="") |
static void | OpenTempFile (TString &base, std::ofstream &fp) |
static void | PrintSplashScreen () |
Prints welcome message and infos on version etc. More... | |
static Bool_t | SearchAndOpenKVFile (const Char_t *name, KVSQLite::database &dbfile, const Char_t *kvsubdir="") |
static Bool_t | SearchAndOpenKVFile (const Char_t *name, std::ifstream &file, const Char_t *kvsubdir="", KVLockfile *locks=0) |
static Bool_t | SearchAndOpenKVFile (const Char_t *name, std::ofstream &file, const Char_t *kvsubdir="", KVLockfile *locks=0) |
static Bool_t | SearchKVFile (const Char_t *name, TString &fullpath, const Char_t *kvsubdir="") |
static const Char_t * | WorkingDirectory () |
Static Public Member Functions inherited from TObject | |
static Longptr_t | GetDtorOnly () |
static Bool_t | GetObjectStat () |
static void | SetDtorOnly (void *obj) |
static void | SetObjectStat (Bool_t stat) |
Public Attributes inherited from TObject | |
kBitMask | |
kCanDelete | |
kCannotPick | |
kHasUUID | |
kInconsistent | |
kInvalidObject | |
kIsOnHeap | |
kIsReferenced | |
kMustCleanup | |
kNoContextMenu | |
kNotDeleted | |
kObjInCanvas | |
kOverwrite | |
kSingleKey | |
kWriteDelete | |
kZombie | |
Protected Attributes inherited from KVDataRepository | |
TString | fAccessprotocol |
TString | fAccessroot |
Bool_t | fCanWrite |
const KVDataSet * | fCommitDataSet |
used by CreateNewFile and CommitFile More... | |
TString | fCommitDataType |
TString | fCommitFileName |
KVDataSetManager * | fDSM |
handles datasets in repository More... | |
TSeqCollection * | fHelpers |
List of helper classes for alternative file/directory access. More... | |
TString | fLocalrootdir |
TString | fReadprotocol |
TString | fReadroot |
TString | fRfiorootdir |
TString | fRfioserver |
TString | fTransferExec |
TString | fTransferserver |
TString | fTransfertype |
TString | fTransferuser |
TString | fXRDtunHost |
Bool_t | fXRDtunnel |
Int_t | fXRDtunPort |
Int_t | fXRDtunRetry |
TString | fXRDtunSpec |
TString | fXRDtunUser |
TString | fXrootdrootdir |
TString | fXrootdserver |
Protected Attributes inherited from KVBase | |
TString | fLabel |
label for the object More... | |
Protected Attributes inherited from TNamed | |
TString | fName |
TString | fTitle |
Protected Attributes inherited from TObject | |
kOnlyPrepStep | |
#include <KVRemoteDataRepository.h>
KVRemoteDataRepository::KVRemoteDataRepository | ( | ) |
Default constructor.
Definition at line 22 of file KVRemoteDataRepository.cpp.
|
virtual |
|
virtual |
Checks if the run file of given type is physically present in dataset subdirectory, i.e. (schematically), if
/root_of_data_repository/[datasetdir]/[datatype]/[runfile]
exists. If it does, the returned value is kTRUE (=1). For a remote data repository, this is not possible. Always returns kFALSE.
Reimplemented from KVDataRepository.
Definition at line 106 of file KVRemoteDataRepository.cpp.
|
virtual |
Returns kTRUE if the following path is valid /root_of_data_repository/dir/[subdir] For a remote data repository, this is not possible. Always returns kFALSE.
Reimplemented from KVDataRepository.
Definition at line 48 of file KVRemoteDataRepository.cpp.
|
virtual |
Copy file [datasetdir]/[datatype]/[filename] from the repository to [destination] We check if the file to copy exists. For remote repositories, this uses the bbftp protocol (if available). NEEDS IMPLEMENTATION.
Reimplemented from KVDataRepository.
Definition at line 132 of file KVRemoteDataRepository.cpp.
|
virtual |
Copy file [source] to [datasetdir]/[datatype]/[filename] in the repository For remote repositories, this uses the bbftp protocol (if available). NEEDS IMPLEMENTATION.
Reimplemented from KVDataRepository.
Definition at line 152 of file KVRemoteDataRepository.cpp.
|
virtual |
Delete repository file [datasetdir]/[datatype]/[filename] Impossible on distant data repositories.
Reimplemented from KVDataRepository.
Definition at line 202 of file KVRemoteDataRepository.cpp.
|
virtual |
Impossible on distant data repositories. Will return 0 (null pointer).
Reimplemented from KVDataRepository.
Definition at line 186 of file KVRemoteDataRepository.cpp.
|
virtual |
Checks if the run file of given type is physically present in dataset subdirectory, i.e. (schematically), if
/root_of_data_repository/[datasetdir]/[datatype]/[runfile]
exists. If it does, the returned value is kTRUE (=1), in which case the FileStat_t object contains information about the file. For a remote data repository, this is not possible. Always returns kFALSE.
Reimplemented from KVDataRepository.
Definition at line 74 of file KVRemoteDataRepository.cpp.
|
virtual |
When accessing remote data repositories, the operating system corresponding to the remote file system may not be the same as the local operating system (corresponding to gSystem). Therefore use of gSystem->ConcatFileName may lead to paths which mix '/' and '\'. We check for this here by converting the result of KVDataRepository::GetFullPath depending on whether the root directory of this repository contains '/' or '\'
Reimplemented from KVDataRepository.
Definition at line 287 of file KVRemoteDataRepository.cpp.
|
virtual |
Used by KVDataTransfer. Returns the full path needed to transfer a runfile belonging to the given dataset either from or to the repository, using sftp or bbftp etc. This is a concatenation of the repository root directory with the dataset subdirectories and filename. However, for a remote data repository, the type of operating system may be different to gSystem, i.e. the system on the local machine. In this case, gSystem->ConcatFileName may not necessarily give the right paths for the remote machine: e.g. local machine is Windows, remote repository is Linux, paths for Linux machine will have a mix of "/" and "\" in them. We try to remedy this by looking at fLocalrootdir: if it contains "/" we take the result of the concatenation performed by KVDataRepository::GetFullPathToTransferFile and replace any "\" by "/", and vice versa.
Reimplemented from KVDataRepository.
Definition at line 242 of file KVRemoteDataRepository.cpp.
|
virtual |
Before reading data in a remote data repository, it may be necessary to 'connect' to the repository in some way: e.g. we may need to open an SSH tunnel to an xrootd server. Call this method to make sure any required connection has been made, and if successful the method returns kTRUE; otherwise return kFALSE.
Configure using the following environment variables:
ccali.DataRepository.XRDTunnel.host: ccali.in2p3.fr ccali.DataRepository.XRDTunnel.port: 10000
The host and port to use are obligatory. Optionally you may also change the following:
ccali.DataRepository.XRDTunnel.retry: #max number of tries before giving up on connection, default = 30
ccali.DataRepository.XRDTunnel.user: #username for login to tunnel host. default is local username.
Reimplemented from KVDataRepository.
Definition at line 332 of file KVRemoteDataRepository.cpp.
|
inlinevirtual |
Returns kTRUE for remote repositories, kFALSE for local repositories.
Reimplemented from KVDataRepository.
Definition at line 76 of file KVRemoteDataRepository.h.
|
virtual |
Create a new subdirectory in the repository Impossible on distant data repositories.
Reimplemented from KVDataRepository.
Definition at line 171 of file KVRemoteDataRepository.cpp.
|
protectedvirtual |
Create and return pointer to new data set manager.
Reimplemented from KVDataRepository.
Definition at line 218 of file KVRemoteDataRepository.cpp.