Metadata-Version: 1.1
Name: ck
Version: 1.6.14
Summary: Collective Knowledge - lightweight knowledge manager to organize, cross-link, share and reuse artifacts
Home-page: https://github.com/ctuning/ck/wiki
Author: Grigori Fursin and non-profit cTuning foundation
Author-email: Grigori.Fursin@cTuning.org
License: BSD 3-clause
Description: Are you an individual researcher or organization performing
        many experiments on a regular basis? You may find the Collective 
        Knowledge framework (CK) useful if you suffer from one or
        more of the following problems:
        
        * instead of innovating, you spend weeks and months preparing 
        ad-hoc experimental workflows, which you either throw away 
        when your ideas are not validated or need to maintain
        (adapting to ever changing software, hardware, interfaces and
        data formats);
        
        * you have trouble sharing whole experimental workflows 
        and results with your colleagues since they use different
        operating systems, tools, libraries and hardware
        (and they do need to use their latest environment rather 
        than possibly outdated Docker or VM images);
        
        * you have trouble managing and reusing your own scripts, tools, 
        data sets and reproducing your own results from past projects;
        
        * you have trouble retrieving data from your own or someone else's
        "black-box" database (particularly if you do not know the schema);
        
        * you spend lots of time updating your reports and papers whenever
        you obtain new results;
        
        * you do not have enough realistic workloads, benchmarks and data sets
        for your research;
        
        * you face the ever increasing number of experimental choices
        to explore in complex design and optimization spaces;
        
        * you accumulate vast amounts of raw experimental data but do not
        know what the data is telling you ("big data" problem);
        
        * you want to extract knowledge from raw data in form of models
        but never find time to master powerful predictive analytics techniques;
        
        * your organization pays dearly for its computational needs
        (in particular, for hardware and energy used in data
        centers and supercomputers) while you suspect they could
        be met at a fraction of the cost (if, for example, your
        deep learning algorithms could run 10 times faster).
        
        Over the past 15 years, we have suffered from all the above
        problems, which intolerably slowed down our own research
        (on developing faster, smaller, more energy efficient and
        reliable computer systems via multi-objective autotuning,
        machine learning and run-time adaptation). 
        Eventually, we have realized that the above problems 
        can only be tackled collaboratively by bringing together
        an interdisciplinary community.
        
        Hence, we designed Collective Knowledge (CK) as 
        just a small and highly customizable Python wrapper
        framework with a unified JSON API, command line, web
        services and meta-descriptions. This allows researchers 
        gradually warp and glue together any existing software, 
        hardware and data, share and reuse wrappers via Git, 
        unify information flow between them, quickly prototype 
        experimental workflows from shared artifacts, 
        apply predictive analytics and enable interactive articles.
        
        CK is an open-source (under permissive license),
        lightweight (< 1 MB) and very portable research SDK. It has
        minimal dependencies and simple interfaces with software
        written in C, C++, Fortran, Java, PHP and other languages.
        Please, check out CK documentation and Getting Started
        Guide for more details: http://github.com/ctuning/ck/wiki
        
        Though seemingly simple, such agile approach already proved
        to be powerful enough to help scientists and research engineers:
        
        * abstract and unify access to their software, hardware and data 
        via CK modules (wrappers) with a simple JSON API 
        while protecting users from continuous low-level changes
        and exposing only minimal information needed for research
        and experimentation (this, in turn, enables simple 
        co-existence of multiple tools and libraries such as 
        different versions of compilers including LLVM, GCC and ICC);
        
        * provide a simple and user-friendly directory structure 
        (CK repositories) to gradually convert all local artifacts 
        (scripts, benchmarks, data sets, tools, results, predictive models, 
        graphs, articles) into searchable, reusable and interconnected 
        CK entries (components) with unique IDs and open JSON-based meta 
        information while getting rid of all hardwired paths;
        
        * quickly prototype research ideas from shared components
        as LEGO(TM), unify exchange of results in schema-free JSON
        format and focus on knowledge discovery (only when idea
        is validated you should spend extra time on adding proper
        types, descriptions and tests, and not vice versa);
        
        * easily share CK repositories with whole experimental
        setups and templates with the community via popular public
        services including GitHub and BitBucket while keeping track
        of all development history;
        
        * speed up search across all your local artifacts by JSON
        meta information using popular ElasticSearch (optional);
        
        * involve the community or workgroups to share realistic
        workloads, benchmarks, data sets, tools, predictive models 
        and features in a unified and customizable format;
        
        * reproduce empirical experimental results in a different environment 
        and under different conditions, and apply statistical analysis 
        (similar to physics) rather than just replicating them 
        - useful to analyze and validate varying results
        (such as performance and energy);
        
        * use built-in CK web server to view interactive graphs
        and articles while easily crowdsourcing experiments 
        using spare computational resources (mobile devices, data centers,
        supercomputers) and reporting back unexpected behavior; 
        
        * obtain help from an interdisciplinary community to explain 
        unexpected behavior when reproducing experiments, solve it 
        by improving related CK modules and entries, and immediately 
        push changes back to the community (similar to Wikipedia);
        
        * simplify the use of statistical analysis and predictive
        analytics techniques for non-specialists via CK modules
        and help you process large amount of experimental results
        (possibly on the fly via active learning), share and
        improve predictive models and features (knowledge), and
        effectively compact "big data".
        
        For example, our colleagues successfully use CK to
        accelerate computer systems' research and tackle issues
        known and unsolved for more than 15 years. They started
        practically enabling customizable, extensible and
        multi-objective software/hardware optimization, run-time
        adaptation and co-design as a CK experimental workflow
        shared via GitHub. The community can now gradually expose
        various tuning choices (algorithm and OpenCL/CUDA/MPI
        parameters, compiler flags, polyhedral transformations,
        CPU/GPU frequency, etc) and objectives (execution time,
        code size, compilation time, energy, processors size,
        accuracy, reliability) . The community can also reuse
        shared autotuning and machine learning plugins to speed
        up exploration of large and non-linear optimization spaces
        and even enable run-time adaptation (self-tuning computer
        systems).
        
        For example, check out public GCC/LLVM optimization results 
        of various shared workloads across  diverse hardware including
        mobile devices provided by volunteers:
        * http://cTuning.org/crowd-results
        * https://play.google.com/store/apps/details?id=openscience.crowdsource.experiments
        
        Furthermore, our colleagues have managed to speed up their real-world
        applications across latest platforms
        (from mobile phones to cloud servers) by 10x with the same
        numerical accuracy, reduce energy by 30% and code size
        by 50%. Furthermore, such CK templates can be easily reused
        in other research scenarios while allowing students and
        researchers start new experiments or reproduce others' results
        in minutes rather than days and weeks, as described here:
        * http://cknowledge.org/interactive-report
        * http://github.com/ctuning/ck/wiki/Getting_started_guide_example_slambench
        * https://github.com/ctuning/ck/wiki/Getting_started_guide_clsmith
        * http:/github.com/ctuning/ck-autotuning
        * http:/github.com/ctuning/reproduce-pamela-project
        * http:/github.com/ctuning/reproduce-carp-project
        * https://github.com/dividiti/gemmbench
        
        Our long-term mission is to help the research community
        dramatically accelerate knowledge discovery via open,
        agile, collaborative and reproducible research,
        experimentation and knowledge sharing while keeping it as
        simple as GitHub and Wikipedia - do join us!
        
        For more details, please see our recent vision paper at DATE'16, 
        ADAPT workshop, artifact sharing and evaluation initiative for 
        computer systems' conferences, live demo with the CK-based
        reproducible and interactive articles, and ARM's testimonials 
        about CK:
        * http://bit.ly/ck-date16
        * http://cTuning.org/ae
        * http://adapt-workshop.org
        * http://cknowledge.org/repo
        * http://es.iet.unipi.it/tetracom/content/index.php/impacts
        
        Documentation 
        =============
        * Full CK documentation (including motivation, getting started guide and tutorials): http://github.com/ctuning/ck/wiki
        * All shared modules: https://github.com/ctuning/ck/wiki/Shared_modules
        * All shared repositories: https://github.com/ctuning/ck/wiki/Shared_repos
        
        License
        =======
        * Permissive 3-clause BSD license (see LICENSE.txt file for more details).
        
        Authors
        =======
        * Grigori Fursin, http://fursin.net
        * Anton Lokhmotov, https://www.hipeac.net/~anton
        
        Copyright
        =========
        (C)opyright 2014-2016 Grigori Fursin, Anton Lokhmotov and contributors
        
        Minimal requirements
        ====================
        * Python >= 2.6 (3.0+ is natively supported)
        * GIT command line client
        
        If you would like to use shared repositories with already implemented
        crowd-benchmarking, multi-objective autotuning, statistical analysis,
        predictive analytics, interactive graphs, etc, you may need to install
        extra Python packages: 
        
        * 'matplotlib','scipy' and 'numpy' to plot graphs and perform statistical 
        analysis (during performance/energy/accuracy autotuning and modeling)
        * 'sklearn-kit' for predictive analytics
        * 'psutil' to properly terminate running process on Windows after timeout 
        (during program autotuning and design space exploration)
        * 'Tkinter' or 'tkinter' or 'pyperclip' if copy to clipboard functionality 
        is required (to reproduce CK experiments from the web)
        
        Alternatively, you can use Anaconda python distribution
        which includes all scientific packages required for CK-based
        collaborative and reproducible experimentation.
        
        You can also considerably speed up local search by installing 
        free third-party Hadoop-based ElasticSearch (http://elastic.co) 
        and turning on transparent indexing in CK.
        
        We successfully validated CK on Linux, Windows, Mac OS X 
        and partially on Android.
        
        Minimal installation
        ====================
        
        Download latest CK from GitHub:
        
         $ git clone https://github.com/ctuning/ck.git ck
        
        Add ck/bin directory to PATH variable
        
         $ export PATH=$PWD/ck/bin:$PATH
        
        On Windows, you should manually add full path to ck/bin 
        to you PATH environment variable.
        
        If you would like to use CK from ipython notebook or other
        python applications, you can install it as a standard 
        Python library (you may need root privileges depending 
        on how your Python is installed):
        
         $ cd ck
        
         $ sudo python setup.py install
            and/or
         $ sudo python3 setup.py install
        
        By default, CK attempts to find and use the latest version of installed Python, 
        i.e. v3 and only then v2. However, this may cause ambiguities with above 
        installation. If you get an error "no module named ck.kernel", you can
        force CK to use a specific Python version by changing environment 
        variable "CK_PYTHON", i.e.:
        
         $ export CK_PYTHON=python3
         
         or
        
         $ export CK_PYTHON=python
        
        Other installation possibilities (via PIP, Conda, Debian) are described
        here: https://github.com/ctuning/ck/wiki/Installation 
        
        Basic usage
        ===========
        
        You can check that CK is installed correctly via:
        
         $ ck
        
        You should see information about CK options.
        
        If you have IPython installed, you can then check CK library 
        installation as following:
        
         $ ipython
        
         $ import ck.kernel as ck
        
         $ ck.test()
        
         $ ck.access('list module')
        
         $ ck.access('load kernel default')
        
        Now, you can use CK to pull various shared CK repositories
        (code and data shared as reusable components via GitHub
        or Bitbucket) and run shared experimental workflows.
        
        For example, we use CK to systematize and accelerate our computer systems' 
        research, perform systematic benchmarking across any existing hardware, 
        crowdsource multi-objective program optimization, share realistic workloads, 
        build predictive models of performance, analyze representative benchmarks 
        and data sets, install missing tools, etc. Rather than wasting weeks and
        months developing your own ad-hoc experimental setups, you can now obtain,
        compile and run any shared benchmark on your Linux machine with GCC 
        in less than a minute via:
        
         $ ck pull repo:ctuning-programs
        
         $ ck compile program:cbench-automotive-susan --speed
        
         $ ck run program:cbench-automotive-susan
        
        You will be asked a few questions and for now you can just press Enter 
        to select the default choices.
        
        Please refer to CK documentation (https://github.com/ctuning/ck/wiki)
        to find out how to compile and run such benchmarks on Windows or Android,
        autotune or crowdtune them (performance, energy, accuracy, size, etc),
        share other realistic workloads, benchmarks and data sets, apply machine 
        learning to explain unexpected behavior or predict optimizations, 
        and quickly prototype your own research ideas.
        
        CK also helps you get rid of hardwired paths for your local data sets,
        tools and scripts - instead you can simply query CK and obtain all 
        shared data sets by specific tags simply via
        
         $ ck search dataset --tags=image,jpeg
        
         $ ck find dataset:[name from above list]
        
        You can also easily add your own CK modules to serve
        as wrappers (containers) with unified API for your data and
        tools while gradually substituting all your ad-hoc scripts.
        For example, you can add your own module 'hello' with
        an action 'say' and with an entry 'world' as following (you
        can select all default values by just pressing Enter):
        
         $ ck add module:hello
        
         $ ck add_action module --func=say
        
         $ ck add hello:world
        
         $ ck say hello:world
        
        You can now find and customize Python dummy function 'say' 
        in the CK module 'hello' (module.py) via
         
         $ ck find module:hello
        
        You can also find associated CK entry hello:say and modify 
        its meta information (.cm/meta.json) via
        
         $ ck find hello:world
        
        Finally, you can start CK web server either to be able
        to collect experimental results from other users or to
        browse your local artifacts (including interactive graphs
        and articles) in a user-friendly way:
        
         $ ck start web
        
         $ open browser with URL http://localhost:3344
        
        Please, check out CK documentation (including Getting Started Guide 
        and various tutorials) for more details:
        
        * https://github.com/ctuning/ck/wiki
        
        Participating in experiment crowdsourcing
        =========================================
        You can easily partipcate in collaborative program optimization and 
        machine learning using the following two steps:
        
         $ ck pull repo:ck-crowdtuning
         
         $ ck crowtune program
        
        All public optimization results are available at http://cTuning.org/crowd-results
        
        Questions/comments/discussions?
        ===============================
        Please, use our mailing lists:
        * Open, collaborative and reproducible R&D including knowledge preservation, sharing and reuse:
          http://groups.google.com/group/collective-knowledge
        * Software and hardware multi-objective (performance/energy/accuracy/size/reliability/cost)
          benchmarking, autotuning, crowdtuning and run-time adaptation: http://groups.google.com/group/ctuning-discussions
        
        Publications
        ============
        Concepts has been described in the following publications:
        
        * http://arxiv.org/abs/1506.06256 (CPC'15)
        * http://bit.ly/ck-date16 (DATE'16)
        * http://cknowledge.org/interactive-report
        * http://hal.inria.fr/hal-01054763 (Journal of Scientific Programming'14)
        * http://arxiv.org/abs/1406.4020 (TRUST'14 @ PLDI'14)
        * https://hal.inria.fr/inria-00436029 (GCC Summit'09)
        
        If you found CK useful and/or interesting, you are welcome
        to reference any of the above publications in your articles
        and reports. You can download above references in BibTex 
        format here:
        
        * https://raw.githubusercontent.com/ctuning/ck-guide-images/master/collective-knowledge-refs.bib
        
        CK-powered projects
        ===================
        
        We currently use and improve CK is the following computer systems'
        research projects:
        
        * building public repository of realistic workloads, benchmarks, 
        data sets, tools, predictive models and optimization knowledge 
        in a unified format with the help of the computer engineering 
        community (http://github.com/ctuning/ctuning-programs , 
        http://github.com/ctuning/reproduce-pamela-project ,
        http://github.com/ctuning/reproduce-carp-project ,
        http://github.com/ctuning/ctuning-datasets-min ,
        http://cknowledge.org/repo , http://c-mind.org/repo)
        
        * implementing universal multi-dimensional,
        multi-objective, plugin-based autotuning combined with
        crowdsourcing, predictive analytics and run-time adaptation
        (to enable self-optimizing computer systems). We support
        OpenCL, CUDA, OpenMP, MPI, compiler and any other tuning
        for performance, energy, accuracy, size, reliability, cost
        and any other metrics across small kernels/codelets and
        large applications (http://github.com/ctuning/ck-autotuning, 
        http://github.com/ctuning/ck-analytics , 
        http://github.com/ctuning/ck-env ,
        http://cknowledge.org/repo/web.php?wcid=29db2248aba45e59:cd11e3a188574d80).
        
        * implementing crowd-benchmarking (crowdsourcing workload characterization
        and compiler heuristic construction across numerous architectures using shared
        computational resources such as mobile phones, tablets, cloud services, etc.
        (https://play.google.com/store/apps/details?id=com.collective_mind.node ,
        http://cknowledge.org/repo/web.php?wcid=29db2248aba45e59:cd11e3a188574d80)
        
        * supporting artifact evaluation initiatives for major conferences
        and journals where all artifacts are shared as reusable components (and not
        just as black box virtual machine images) along with publications
        (http://cTuning.org/ae , http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 ,
        http://arxiv.org/abs/1406.4020 , http://adapt-workshop.org)
        
        * enabling open, collaborative and reproducible research and experimentation
        with interactive publications focusing on computer engineering
        ( http://cTuning.org/reproducibility-wiki )
        
        * enabling interactive and reproducible articles for Digital Libraries
        (http://cknowledge.org/repo/web.php?wcid=29db2248aba45e59:cd11e3a188574d80 ,
        http://cknowledge.org/repo/web.php?wcid=29db2248aba45e59:6f40bc99c4f7df58)
        
        * serving as a personal knowledge manager to organize, interconnect
        and preserve all personal coda and data via simple JSON
        meta with UIDs and semantic tags.
        
        You are welcome to add your own CK-powered project here!
        
        Acknowledgments
        ===============
        
        CK development is coordinated by the non-profit cTuning foundation
        (cTuning.org). We would like to thank the EU TETRACOM 609491 project
        (www.tetracom.eu) for initial funding and dividiti (www.dividiti.com)
        for continuing support. We are also extremely grateful to all volunteers
        for their valuable feedback and contributions.
        
Platform: UNKNOWN
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Environment :: Plugins
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: End Users/Desktop
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development
Classifier: Topic :: System
Classifier: Topic :: System :: Benchmark
Classifier: Topic :: Education
Classifier: Topic :: Home Automation
Classifier: Topic :: Adaptive Technologies
Classifier: Topic :: Database
Classifier: Topic :: Utilities
