Base class for transferring data between data repositories.
If the user has defined two or more repositories in her $HOME/.kvrootrc file, e.g. as in the example #3 given in $KVROOT/KVFiles/.kvrootrc:
#Case 3: user has data files on his machine or on a locally-mounted disk, and can access a remote data repository using xrootd via an SSH tunnel on port 10000 (the example given #is for the xrootd server at ccali centre de calcul).
#definition of local repository DataRepository: default default.DataRepository.Type: local default.DataRepository.RootDir: /home/frankland/Data
#definition of remote repository +DataRepository: ccali ccali.DataRepository.Type: remote ccali.DataRepository.RootDir: cchpssindra:/hpss/in2p3.fr/group/indra ccali.DataRepository.ReadProtocol: root ccali.DataRepository.XRDServer: localhost:10000 ccali.DataRepository.XRDRootDir: /hpss/in2p3.fr/group/indra ccali.DataRepository.RemoteAvailableRuns.protocol: curl ccali.DataRepository.RemoteAvailableRuns.url: http://indra.in2p3.fr/KaliVedaDoc ccali.DataRepository.FileTransfer.type: bbftp ccali.DataRepository.FileTransfer.server: ccbbftp.in2p3.fr if your login name is not the same as for ccali, give it here ccali.DataRepository.FileTransfer.user:
then it is possible to transfer files from one to the other by running one of the following commands on the local machine:
KVDataTransfer::NewTransfer("ccali", "default")->Run() // transfer from ccali to local machine KVDataTransfer::NewTransfer("default", "ccali")->Run() // transfer from local machine to ccali
The same menu-driven approach as for submitting analysis tasks is used to allow the user to choose among the available datasets, systems and runs to transfer. The transferred files are copied in to the target repository, creating any missing directories which may be needed.
It is also possible to use a KVDataTransfer object interactively, in this case you should use:
KVDataTransfer* trans = KVDataTransfer::NewTransfer("ccali", "default"); trans->SetDataSet(...); trans->SetDataType(...); trans->SetRuns(...); trans->TransferRuns();
To use the BBFTP transfer facility, a bbftp client must be installed on the user's machine If the client is not located in the user's PATH, he should give the full path to the executable in the configuration, e.g. on WinXP with cygwin version of bbftp client:
ccali.DataRepository.FileTransfer.type: c:\cygwin\bin\bbftp.exe
Definition at line 75 of file KVDataTransfer.h.
Public Member Functions | |
| KVDataTransfer () | |
| virtual | ~ KVDataTransfer () |
| virtual void | Run () |
| virtual void | TransferRuns () |
 Public Member Functions inherited from KVDataSetAnalyser | |
| KVDataSetAnalyser () | |
| Default constructor. More... | |
| virtual | ~KVDataSetAnalyser () |
| Destructor. More... | |
| virtual void | AddJobDescriptionList (TList *) |
| void | ChooseDataSet () |
| void | ChooseDataType () |
| void | ChooseRuns (KVDBSystem *system=nullptr, const Char_t *data_type="") |
| void | ChooseSystem (const Char_t *data_type="") |
| void | ClearRunList () |
| TString | ExpandAutoBatchName (const Char_t *format) const |
| KVDBRun * | GetAnalysedRun () const |
| KVDBSystem * | GetAnalysedSystem () const |
| const KVDataSet * | GetDataSet () const |
| const KVNumberList & | GetFullRunList () const |
| const KV2Body * | GetKinematics () const |
| Return pointer to kinematics of analysed system if defined. More... | |
| Int_t | GetNumberOfFilesToAnalyse () const |
| const Char_t * | GetRecognisedAutoBatchNameKeywords () const |
| KVString | GetRootDirectoryOfDataToAnalyse () const |
| Returns path to data to be analysed. More... | |
| const KVNumberList & | GetRunList () const |
| Int_t | GetRunNumberFromFileName (const Char_t *) |
| Get the run number from the filename. More... | |
| const KVDBSystem * | GetSystem () const |
| Bool_t | IsChooseDataSet () const |
| Bool_t | IsChooseRuns () const |
| Bool_t | IsChooseSystem () const |
| Bool_t | IsChooseTask () const |
| Bool_t | ReadBatchEnvFile (const Char_t *filename) |
| void | Reset () |
| void | SetChooseDataSet (Bool_t yes=kTRUE) |
| void | SetChooseRuns (Bool_t yes=kTRUE) |
| void | SetChooseSystem (Bool_t yes=kTRUE) |
| void | SetChooseTask (Bool_t yes=kTRUE) |
| void | SetDataSet (const Char_t *name) |
| void | SetDataSet (KVDataSet *ds) |
| void | SetFullRunList (const KVNumberList &nl) |
| void | SetRuns (const KVNumberList &nl, Bool_t check=kTRUE) |
| void | SetSystem (KVDBSystem *syst) |
| Set the System used in the analysis. More... | |
| virtual void | SetTree (TTree *) |
| TString | SystemBatchName () const |
| void | WriteBatchEnvFile (const Char_t *jobname, Bool_t save=kTRUE) |
| Public Member Functions inherited from KVDataAnalyser | |
| KVDataAnalyser () | |
| Default constructor. More... | |
| virtual | ~ KVDataAnalyser () |
| Bool_t | BatchMode () const |
| virtual Bool_t | CheckStatusUpdateInterval (Long64_t nevents) const |
| virtual Bool_t | CheckTaskVariables () |
| void | ChooseNbEventToRead () |
| Ask user to set number of events to read. More... | |
| void | ChooseRunningMode () |
| void | ChooseUserClass () |
| Choose the user's analysis class. More... | |
| void | DeleteBatchStatusFile () const |
| Delete batch status file (and backup - '.bak') for batch job. More... | |
| void | DoStatusUpdate (Long64_t nevents) const |
| Print infos on events treated, disk usage, memory usage. More... | |
| Bool_t | DoUserClassFilesExist () |
| KVDataAnalysisTask * | GetAnalysisTask () const |
| const Char_t * | GetBatchName () |
| const Char_t * | GetBatchStatusFileName () const |
| Returns full path to file used to store status of running batch jobs. More... | |
| virtual const KVBatchSystem * | GetBatchSystem () |
| const KVString & | GetDataType () const |
| const Char_t * | GetLaunchDirectory () const |
| Long64_t | GetNbEventToRead (void) const |
| KVDataAnalyser * | GetParent () const |
| EProofMode | GetProofMode () const |
| Double_t | GetStatusUpdateInterval () const |
| virtual Long64_t | GetTotalEntriesToRead () const |
| const Char_t * | GetUserClass () |
| const KVString & | GetUserClassDec () const |
| const KVString & | GetUserClassImp () const |
| const KVString & | GetUserClassOptions () const |
| const KVString & | GetUserIncludes () const |
| const KVString & | GetUserLibraries () const |
| Bool_t | IsMenus () const |
| Bool_t | IsQuit () const |
| Bool_t | IsSubmit () const |
| Bool_t | IsUserClassValid () const |
| virtual void | postAnalysis () |
| virtual void | postEndAnalysis () |
| virtual void | postEndRun () |
| virtual void | postInitAnalysis () |
| virtual void | postInitRun () |
| virtual void | preAnalysis () |
| virtual void | preEndAnalysis () |
| virtual void | preEndRun () |
| virtual void | preInitAnalysis () |
| virtual void | preInitRun () |
| virtual void | RegisterUserClass (TObject *) |
| virtual void | RunMenus () |
| Run data analyser in menu-driven mode. More... | |
| Bool_t | RunningInLaunchDirectory () |
| void | SetAnalysisTask (KVDataAnalysisTask *at) |
| virtual void | SetAuxFileList (TList *) |
| void | SetBatchMode (Bool_t on=kTRUE) |
| void | SetBatchName (const Char_t *batchname) |
| void | SetBatchSystem (KVBatchSystem *bs) |
| void | SetDataType (const Char_t *name) |
| virtual void | SetFileList (TList *) |
| void | SetMenus (Bool_t on=kTRUE) |
| void | SetNbEventToRead (Long64_t nb=0) |
| void | SetParent (KVDataAnalyser *da) |
| void | SetProofMode (EProofMode e) |
| void | SetQuit (Bool_t yes=kTRUE) |
| void | SetStatusUpdateInterval (double t) |
| void | SetSubmit (Bool_t yes=kTRUE) |
| virtual void | SetTriggerConditionsForRun (int) |
| void | SetUserClass (const Char_t *kvs, Bool_t check=kTRUE) |
| void | SetUserClassOptions (const Char_t *o="") |
| void | SetUserIncludes (const Char_t *incDirs=0) |
| void | SetUserLibraries (const Char_t *libs=0) |
| void | UpdateBatchStatusFile (Int_t totev, Int_t evread, TString disk) const |
| Update infos in batch status file. More... | |
| void | UseBaseClassSubmitTask () |
| void | WriteBatchInfo (TTree *) |
| Public Member Functions inherited from TObject | |
| TObject () | |
| TObject (const TObject &object) | |
| virtual | ~TObject () |
| void | AbstractMethod (const char *method) const |
| virtual void | AppendPad (Option_t *option="") |
| virtual void | Browse (TBrowser *b) |
| ULong_t | CheckedHash () |
| virtual const char * | ClassName () const |
| virtual void | Clear (Option_t *="") |
| virtual TObject * | Clone (const char *newname="") const |
| virtual Int_t | Compare (const TObject *obj) const |
| virtual void | Copy (TObject &object) const |
| virtual void | Delete (Option_t *option="") |
| virtual Int_t | DistancetoPrimitive (Int_t px, Int_t py) |
| virtual void | Draw (Option_t *option="") |
| virtual void | DrawClass () const |
| virtual TObject * | DrawClone (Option_t *option="") const |
| virtual void | Dump () const |
| virtual void | Error (const char *method, const char *msgfmt,...) const |
| virtual void | Execute (const char *method, const char *params, Int_t *error=0) |
| virtual void | Execute (TMethod *method, TObjArray *params, Int_t *error=0) |
| virtual void | ExecuteEvent (Int_t event, Int_t px, Int_t py) |
| virtual void | Fatal (const char *method, const char *msgfmt,...) const |
| virtual TObject * | FindObject (const char *name) const |
| virtual TObject * | FindObject (const TObject *obj) const |
| virtual Option_t * | GetDrawOption () const |
| virtual const char * | GetIconName () const |
| virtual const char * | GetName () const |
| virtual char * | GetObjectInfo (Int_t px, Int_t py) const |
| virtual Option_t * | GetOption () const |
| virtual const char * | GetTitle () const |
| virtual UInt_t | GetUniqueID () const |
| virtual Bool_t | HandleTimer (TTimer *timer) |
| virtual ULong_t | Hash () const |
| Bool_t | HasInconsistentHash () const |
| virtual void | Info (const char *method, const char *msgfmt,...) const |
| virtual Bool_t | InheritsFrom (const char *classname) const |
| virtual Bool_t | InheritsFrom (const TClass *cl) const |
| virtual void | Inspect () const |
| void | InvertBit (UInt_t f) |
| virtual Bool_t | IsEqual (const TObject *obj) const |
| virtual Bool_t | IsFolder () const |
| R__ALWAYS_INLINE Bool_t | IsOnHeap () const |
| virtual Bool_t | IsSortable () const |
| R__ALWAYS_INLINE Bool_t | IsZombie () const |
| virtual void | ls (Option_t *option="") const |
| void | MayNotUse (const char *method) const |
| virtual Bool_t | Notify () |
| void | Obsolete (const char *method, const char *asOfVers, const char *removedFromVers) const |
| void | operator delete (void *ptr) |
| void | operator delete[] (void *ptr) |
| void * | operator new (size_t sz) |
| void * | operator new (size_t sz, void *vp) |
| void * | operator new[] (size_t sz) |
| void * | operator new[] (size_t sz, void *vp) |
| TObject & | operator= (const TObject &rhs) |
| virtual void | Paint (Option_t *option="") |
| virtual void | Pop () |
| virtual void | Print (Option_t *option="") const |
| virtual Int_t | Read (const char *name) |
| virtual void | RecursiveRemove (TObject *obj) |
| void | ResetBit (UInt_t f) |
| virtual void | SaveAs (const char *filename="", Option_t *option="") const |
| virtual void | SavePrimitive (std::ostream &out, Option_t *option="") |
| void | SetBit (UInt_t f) |
| void | SetBit (UInt_t f, Bool_t set) |
| virtual void | SetDrawOption (Option_t *option="") |
| virtual void | SetUniqueID (UInt_t uid) |
| virtual void | SysError (const char *method, const char *msgfmt,...) const |
| R__ALWAYS_INLINE Bool_t | TestBit (UInt_t f) const |
| Int_t | TestBits (UInt_t f) const |
| virtual void | UseCurrentStyle () |
| virtual void | Warning (const char *method, const char *msgfmt,...) const |
| virtual Int_t | Write (const char *name=0, Int_t option=0, Int_t bufsize=0) |
| virtual Int_t | Write (const char *name=0, Int_t option=0, Int_t bufsize=0) const |
Static Public Member Functions | |
| static KVDataTransfer * | NewTransfer (const Char_t *source_rep, const Char_t *target_rep) |
| Static Public Member Functions inherited from KVDataAnalyser | |
| static Bool_t | AbortProcessingLoop () |
| static KVDataAnalyser * | GetAnalyser (const Char_t *plugin) |
| static Bool_t | IsRunningBatchAnalysis () |
| static void | RunAnalyser (const Char_t *plugin="") |
| static void | SetAbortProcessingLoop (Bool_t now=kTRUE) |
| Static Public Member Functions inherited from TObject | |
| static Longptr_t | GetDtorOnly () |
| static Bool_t | GetObjectStat () |
| static void | SetDtorOnly (void *obj) |
| static void | SetObjectStat (Bool_t stat) |
Protected Attributes | |
| TString | fCmdFile |
| name of command file given to transfer agent More... | |
| Bool_t | fOK |
| may be set to kFALSE by init(), in which case Run() will abort More... | |
| KVDataRepository * | fSourceRep |
| repository containing source files More... | |
| KVDataRepository * | fTargetRep |
| repository where files will be copied More... | |
| TString | fTransferExec |
| full path to transfer client executable More... | |
| Protected Attributes inherited from KVDataAnalyser | |
| TList * | fWorkDirInit |
| list of files in working directory before task runs More... | |
| Protected Attributes inherited from TObject | |
| kOnlyPrepStep | |
Additional Inherited Members | |
| Public Types inherited from KVDataAnalyser | |
| enum | EProofMode { None , Lite , Proof } |
| Public Types inherited from TObject | |
| enum | EDeprecatedStatusBits |
| enum | EStatusBits |
| Public Attributes inherited from TObject | |
| kBitMask | |
| kCanDelete | |
| kCannotPick | |
| kHasUUID | |
| kInconsistent | |
| kInvalidObject | |
| kIsOnHeap | |
| kIsReferenced | |
| kMustCleanup | |
| kNoContextMenu | |
| kNotDeleted | |
| kObjInCanvas | |
| kOverwrite | |
| kSingleKey | |
| kWriteDelete | |
| kZombie | |
#include <KVDataTransfer.h>
Constructor & Destructor Documentation
◆ KVDataTransfer()
| KVDataTransfer::KVDataTransfer | ( | ) |
Default constructor. Use KVDataTransfer::NewTransfer to create a new data transfer object with the correct properties to transfer files between two repositories.
Definition at line 28 of file KVDataTransfer.cpp.
◆ ~ KVDataTransfer()
|
virtual |
Member Function Documentation
◆ CheckTargetRepository()
|
protectedvirtual |
Make sure that the target repository has the necessary dataset/subdirectory to receive the transferred files. If not, we create the new dataset/subdir.
Definition at line 257 of file KVDataTransfer.cpp.
◆ ChooseAnalysisTask()
|
inlineprotectedvirtual |
Print list of all possible data analysis tasks for the chosen data set and invite the user to choose one of them. If only one task available, it is automatically selected not possible in batch mode
Reimplemented from KVDataSetAnalyser.
Definition at line 86 of file KVDataTransfer.h.
◆ ExecuteCommand()
|
protectedpure virtual |
Implemented in KVDataTransferXRD.
◆ init()
Initialisation of data transfer.
Definition at line 286 of file KVDataTransfer.cpp.
◆ NewTransfer()
|
static |
Creates a new data transfer object to transfer files between the two named data repositories. The type of the created object depends on the values of the environment variables (defined in .kvrootrc file):
source_rep.DataRepository.FileTransfer.type target_rep.DataRepository.FileTransfer.type
If either of these = bbftp, we create a KVDataTransferBBFTP object. If source_rep.DataRepository.FileTransfer.type = xrd, we create a KVDataTransferXRD object. By default (no type given), we use a KVDataTransferSFTP object. In fact, the type of object created is defined in .kvrootrc by the following plugins:
Plugins for data transfer between repositories
Used by KVDataTransfer::NewTransfer
Plugin.KVDataTransfer: sftp KVDataTransferSFTP KVMultiDet "KVDataTransferSFTP()" +Plugin.KVDataTransfer: bbftp KVDataTransferBBFTP KVMultiDet "KVDataTransferBBFTP()" +Plugin.KVDataTransfer: xrd KVDataTransferXRD KVMultiDet "KVDataTransferXRD()"
Definition at line 70 of file KVDataTransfer.cpp.
◆ Run()
|
virtual |
Perform file transfer The user selects runs which are available in the source repository in the same way as when performing data analysis Note that this will make the 'source' repository the current active repository (i.e. gDataRepository and gDataSetManager will correspond to source repository after execution of this command).
Reimplemented from KVDataAnalyser.
Definition at line 149 of file KVDataTransfer.cpp.
◆ set_dataset_name()
Set dataset to be analysed. If 'name' is not the name of a valid and available dataset in the 'source' data repository an error message is printed.
Reimplemented from KVDataSetAnalyser.
Definition at line 299 of file KVDataTransfer.cpp.
◆ set_dataset_pointer()
Set dataset to be used for transfer. If the chosen dataset is not available, an error message is printed Only datasets which are available in the source repository can be transferred If the pointer actually corresponds to a dataset in the target repository, we replace it with a pointer to the dataset with the same name in the source repository. allow user to reset dataset pointer to 0
Reimplemented from KVDataSetAnalyser.
Definition at line 327 of file KVDataTransfer.cpp.
◆ SetTransferExec()
Definition at line 82 of file KVDataTransfer.h.
◆ SubmitTask()
|
inlineprotectedvirtual |
In interactive mode, the data analysis task is performed by instanciating and initialising the KVDataAnalyser child class specified by the task, and then calling its Run() method. In batch mode, the job is submitted to the chosen batch system.
Reimplemented from KVDataAnalyser.
Definition at line 89 of file KVDataTransfer.h.
◆ TransferRuns()
|
virtual |
Based on information gathered from user (see Run()), perform the transfer of files. Any missing directories in the target repository are created beforehand. After transfer, the available runlist for the target repository is updated
Definition at line 205 of file KVDataTransfer.cpp.
◆ WriteTransferScript()
|
protectedpure virtual |
Implemented in KVDataTransferXRD.
Member Data Documentation
◆ fCmdFile
|
protected |
name of command file given to transfer agent
Definition at line 79 of file KVDataTransfer.h.
◆ fOK
|
protected |
may be set to kFALSE by init(), in which case Run() will abort
Definition at line 99 of file KVDataTransfer.h.
◆ fSourceRep
|
protected |
repository containing source files
Definition at line 77 of file KVDataTransfer.h.
◆ fTargetRep
|
protected |
repository where files will be copied
Definition at line 78 of file KVDataTransfer.h.
◆ fTransferExec
|
protected |
full path to transfer client executable
Definition at line 80 of file KVDataTransfer.h.
The documentation for this class was generated from the following files:
- kaliveda.doxygen/KVMultiDet/analysis/KVDataTransfer.h
- kaliveda.doxygen/KVMultiDet/analysis/KVDataTransfer.cpp
