Posts

Posts

  • Jean-Pierre Dijcks
    BDA 4.14 - Current plan and timelines to uptake CDH 5.1645.0
    Topic posted October 12, 2018 by Jean-Pierre DijcksSilver Medal: 2,000+ Points, tagged Big Data Appliance, Hadoop in Data Management > Big Data public
    Title:
    BDA 4.14 - Current plan and timelines to uptake CDH 5.16
    Summary:
    Roadmap to the last CDH 5.x release on BDA and BDCS/BDCC
    Content:

    A quick update to the community on the current estimated timelines around CDH 5.16 on BDA (and the other platforms).

    While we are working on up taking, and enabling an easy migration to C6.0.1, we are also working on getting the last 5.x release to everyone. The current timelines for BDA 4.14 are around the early - mid December timeframe. That is going to be based on the then-current 5.16 version, most likely something like 5.16.1.

    Note that this is Cloudera's last "major" C5 release (e.g. 5.X). Various maintenance releases (e.g. 5.x.Y) will be arriving and those are of course supported per our standard policies.

    For those moving to the latest 5.x releases do keep the migration to C6 in mind. Review this link for this from the Cloudera side of things: https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_install_upgrade_notes.html#concept_gzr_xgv_pdb. Keep an eye of these notes and the release notes for 6.0.1 once it arrives, as this does impact your strategy to move between releases.

    Version:
    BDA 4.14, CDH 5.16, CDH 6.0
  • Marty Gubar
    Big Data Appliance 5.1 now available
    Topic posted November 14, 2019 by Marty GubarBlue Ribbon: 750+ Points, tagged Big Data Appliance, Hadoop, Spark in Data Management > Big Data public
    Title:
    Big Data Appliance 5.1 now available
    Summary:
    BDA 5.1 available based on Cloudera Enterprise 6.2.1
    Content:

    We are excited to announce that Oracle Big Data Appliance 5.1 is now publicly available.  This is a massive release with major new features across the stack.  It is based on Cloudera Enterprise 6.2.1 and Oracle NoSQL Database 19.3.2 - allowing our customers to take advantage of the latest Apache Hadoop, Spark, Hive and other core Hadoop and Cloudera Enterprise Data Hub features.  

    Here are key details about the release.

    • Based on Cloudera Enterprise Data Hub (6.2.1)
    • Provides an optimized Spark configuration that has shown 50% to 70x performance gains on Oracle Global Leaders’ customer workloads
    • Includes automated installation of Big Data SQL 4.0
    • In-Place Procedure for Migrating Oracle Linux 6 Clusters to Oracle Linux 7
    • Improved security out of the box
    • Support for all Cloudera advanced security options

    Happy installs and upgrades!

    Image:
  • Marty Gubar
    Big Data SQL 3.2 - Cloudera Certified5.0
    Topic posted October 29, 2018 by Marty GubarBlue Ribbon: 750+ Points, tagged Analytics, Big Data Appliance, Big Data SQL, Data Management, Hadoop, SQL, Tip in Data Management > Big Data public
    Title:
    Big Data SQL 3.2 - Cloudera Certified
    Summary:
    Big Data SQL 3.2.1 is now officially certified with Cloudera 5.14.1
    Content:

    Big Data SQL 3.2.1 is now officially Cloudera Certified (see Oracle certified products here).  The certification took place with Cloudera 5.14.1.  The testing captures real world functional integration usage of the product on a cluster that meets certain security requirements, including:

    • Kerberos
    • Sentry
    • HDFS Encryption
    • Auditing
    • Network Encryption. 

    It also examines the product architecture, including integration with Cloudera Manager and key CDH Components.  Finally, it reviews how the product leverages scale-out processing on Hadoop clusters.

  • Marty Gubar
    Big Data SQL 3.2.1.2 for Oracle Database 12.2 Jan Release...5.0
    Topic posted March 15, 2019 by Marty GubarBlue Ribbon: 750+ Points, tagged Big Data Appliance, Big Data SQL, Data Management, Hadoop in Data Management > Big Data public
    Title:
    Big Data SQL 3.2.1.2 for Oracle Database 12.2 Jan Release Update
    Summary:
    BDSQL Patch now available for Oracle Database JAN Release Update
    Content:

    Wanted to give a quick update.

    Big Data SQL 3.2.1.2 and its associated one-off patch for Oracle Database 12.2 January RU are now available for download:

    • Patch 26170659 for DATABASE JAN 2019 RELEASE UPDATE 12.2.0.1.190115.  The one off patch must be applied to both Grid Infrastructure and Database Homes (see patch readme for details).
    • Big Data SQL 3.2.1.2 released on ARU as Patch 29489551: BDSQL 3.2.1.2 INSTALLER BUNDLE.  It will also be made available on edelivery shortly.  This installation must be applied to the Hadoop cluster as it contains updates to the Big Data SQL cells.

    Thanks!

    Marty

  • Marty Gubar
    Big Data SQL 4.0 is now Available5.0
    Topic posted July 11, 2019 by Marty GubarBlue Ribbon: 750+ Points, tagged Analytics, Big Data Appliance, Big Data SQL, Data Management, Hadoop in Data Management > Big Data public
    Title:
    Big Data SQL 4.0 is now Available
    Summary:
    The latest release of Big Data SQL is available for download
    Content:

    I’m excited to announce the availability of Big Data SQL 4.0.  This release is packed with new capabilities:

    Big Data SQL 4.0 can be downloaded from http://edelivery.oracle.com.  It requires Oracle Database 18c (18.6) plus a one off patch (patch # 29379046) that you can download from My Oracle Support.  If you are still on Oracle Database 12.1 or 12.2 – you should continue to use Big Data SQL 3.2.x.

  • Pavel Brunarskiy
    BigDataSQL memory consumption
    Topic posted October 25, 2019 by Pavel Brunarskiy, tagged Big Data SQL, Hadoop in Data Management > Big Data public
    Title:
    BigDataSQL memory consumption
    Summary:
    How to limit BigDataSQL shared memory consumption ?
    Content:

    We tried to install BD SQL 4.0 at CDH cluster that have about 642 Gb of memory at each Data Node. At installation process fails when try to start BD_CELL with failure:

     

    Failed to allocate EXTRA SysV segment of 248545 MB, exceeding system SHMALL limit of 197923328 pages (773138 MB) or SMMNI limit of 4096 segments.

    [RS] Monitoring process /opt/oracle/cell/cellsrv/bin/bdsqlrsomt (pid: 3755286) returned with error: 161

    Errors in file /opt/oracle/cell/log/diag/bdsql/cell/amaterasu2/trace/bdsqlsrvtrc_3755288_main.trc  (incident=185):

    ORA-00600: internal error code, arguments: [main::ocl_lib_serv_init2], [30], [Shared memory create failure], [28], [No space left on device], [ocl_shmem.c], [1456], [], [], [], [], []

     

    We try to set 

    "memory" : {
            "min_hard_limit" : 16384,
            "max_percentage" : 30
    }

    at bds-config.json but it's not helped us. 

    Is it any way to reduce BDS memory consumption?

    Version:
    Big Data SQL 4.0
  • Jean-Pierre Dijcks
    Cloudera 6 released - when will we have it on BDCS/BDCC &...4.5
    Topic posted August 31, 2018 by Jean-Pierre DijcksSilver Medal: 2,000+ Points, tagged Big Data Appliance, Data Management, Hadoop, Spark, SQL in Data Management > Big Data public
    Title:
    Cloudera 6 released - when will we have it on BDCS/BDCC & BDA
    Summary:
    Quick update on Cloudera 6 on BDCS and BDA as well as BDCC
    Content:

    With the release of C6 on August 30th, we are now working towards releasing this on the Oracle Big Data stack (cloud and on-premises).

    We did want to ensure that we communicate some of the important functionality bits and when what would arrive, purely based on the Cloudera schedule:

    (1) The current 6.0 version does not really support Erasure Coding yet. This is because, while the HDFS service supports it, the query engines are not yet running on Erasure Coding. Consequently, you will only be able to do some testing with Erasure coding in terms of cold storage or experimenting with the sizes and reduction in sizes you could achieve. Erasure coding will arrive fully in C6.1.

    (2) Almost all major components in C6 will do a major version uptick. So, rather then wait for C6.1, Oracle will uptake 6.0.1 on BDA, which will enable our customers to move to these newer versions of the components and make use of the new functionality asap on the Oracle Big Data platforms. This will include an upgrade from 5.x to 6.0.1.

    Stay tuned for a more formal update with details on migration, versions, dates etc. which we will post here and on our blog in the next week or so.

  • Jean-Pierre Dijcks
    How do I back-up or ingest data in Big Data Cloud Service?
    Topic posted August 1, 2018 by Jean-Pierre DijcksSilver Medal: 2,000+ Points, tagged Analytics, Big Data Appliance, Data Management, Hadoop, Tip in Data Management > Big Data public
    Title:
    How do I back-up or ingest data in Big Data Cloud Service?
    Summary:
    Ingesting and backing up data to object storage is an important task shown in the step-by-step example
    Content:

    As you are working with Big Data Cloud Service, two common tasks with a common denominator crop up:

    • Ingesting data from the outside world into BDCS
    • Backing up critical data elements to enable restoring in case of issues

    Both of these have a common denominator in that you would probably want to look at Object Storage as a good medium for both ingest and potentially back-up. To show folks how this works, have a look at the following Oracle By Example step-by-step tutorial:

    Ingesting and backing up Data

    Comments welcome of course and let us know if there are questions.

  • candan pehlivanoglu
    How should we gather external table stats on bda via bigdata...15.0
    Topic posted January 2, 2019 by candan pehlivanogluGreen Ribbon: 100+ Points, tagged Analytics, Big Data Appliance, Big Data SQL, Data Management, Hadoop in Data Management > Big Data public
    Title:
    How should we gather external table stats on bda via bigdata sql.
    Summary:
    Gather stats for external tables.
    Content:

    Hello all;

    We use bds for our external tables. We have daily jobs  to ingest these tables on big data and nealry 200 tables. These tables are partitioned on hive. But, on oracle site, since there isn't any partition column, we should gather all statistic every day. It is taking too much time. Do you have any other method for this operation. Since we use these tables on oracle, Oracle should be able to know statistic information.

     

    thank you.

     

  • Jean-Pierre Dijcks
    How to: Automating Data Ingest to Zeppeling Notebooks
    Topic posted August 14, 2018 by Jean-Pierre DijcksSilver Medal: 2,000+ Points, tagged Big Data Appliance, Data Management, Hadoop, Python, Spark, SQL, Tip, Zeppelin in Data Management > Big Data public
    Title:
    How to: Automating Data Ingest to Zeppeling Notebooks
    Summary:
    Leverage data pipelines and big data manager to automate ingest to analytics
    Content:

    In this 20-minute tutorial, you learn how to create and run a new Oracle Big Data Manager pipeline that contains Data Copy and Data Extract jobs. You also import a note into Oracle Big Data Manager that displays the copied and extracted data. 

    http://www.oracle.com/webfolder/technetwork/tutorials/obe/cloud/bigdata/working-with-oracle-bdm-analytic-pipelines/working-with-oracle-bdm-analytic-pipelines.html

    This Hands-on exercise - for those on BDA, BDCS or BDCC - leverages one the latest features added into Big Data Manager.