Posts

Posts

  • 1-8 of 8
  • Pavel Brunarskiy
    BigDataSQL memory consumption
    Topic posted October 25, 2019 by Pavel Brunarskiy, tagged Big Data SQL, Hadoop in Data Management > Big Data public
    Title:
    BigDataSQL memory consumption
    Summary:
    How to limit BigDataSQL shared memory consumption ?
    Content:

    We tried to install BD SQL 4.0 at CDH cluster that have about 642 Gb of memory at each Data Node. At installation process fails when try to start BD_CELL with failure:

     

    Failed to allocate EXTRA SysV segment of 248545 MB, exceeding system SHMALL limit of 197923328 pages (773138 MB) or SMMNI limit of 4096 segments.

    [RS] Monitoring process /opt/oracle/cell/cellsrv/bin/bdsqlrsomt (pid: 3755286) returned with error: 161

    Errors in file /opt/oracle/cell/log/diag/bdsql/cell/amaterasu2/trace/bdsqlsrvtrc_3755288_main.trc  (incident=185):

    ORA-00600: internal error code, arguments: [main::ocl_lib_serv_init2], [30], [Shared memory create failure], [28], [No space left on device], [ocl_shmem.c], [1456], [], [], [], [], []

     

    We try to set 

    "memory" : {
            "min_hard_limit" : 16384,
            "max_percentage" : 30
    }

    at bds-config.json but it's not helped us. 

    Is it any way to reduce BDS memory consumption?

    Version:
    Big Data SQL 4.0
  • Marty Gubar
    Big Data SQL 4.0 is now Available5.0
    Topic posted July 11, 2019 by Marty GubarBlue Ribbon: 750+ Points, tagged Analytics, Big Data Appliance, Big Data SQL, Data Management, Hadoop in Data Management > Big Data public
    Title:
    Big Data SQL 4.0 is now Available
    Summary:
    The latest release of Big Data SQL is available for download
    Content:

    I’m excited to announce the availability of Big Data SQL 4.0.  This release is packed with new capabilities:

    Big Data SQL 4.0 can be downloaded from http://edelivery.oracle.com.  It requires Oracle Database 18c (18.6) plus a one off patch (patch # 29379046) that you can download from My Oracle Support.  If you are still on Oracle Database 12.1 or 12.2 – you should continue to use Big Data SQL 3.2.x.

  • Marty Gubar
    Big Data SQL 3.2.1.2 for Oracle Database 12.2 Jan Release...5.0
    Topic posted March 15, 2019 by Marty GubarBlue Ribbon: 750+ Points, tagged Big Data Appliance, Big Data SQL, Data Management, Hadoop in Data Management > Big Data public
    Title:
    Big Data SQL 3.2.1.2 for Oracle Database 12.2 Jan Release Update
    Summary:
    BDSQL Patch now available for Oracle Database JAN Release Update
    Content:

    Wanted to give a quick update.

    Big Data SQL 3.2.1.2 and its associated one-off patch for Oracle Database 12.2 January RU are now available for download:

    • Patch 26170659 for DATABASE JAN 2019 RELEASE UPDATE 12.2.0.1.190115.  The one off patch must be applied to both Grid Infrastructure and Database Homes (see patch readme for details).
    • Big Data SQL 3.2.1.2 released on ARU as Patch 29489551: BDSQL 3.2.1.2 INSTALLER BUNDLE.  It will also be made available on edelivery shortly.  This installation must be applied to the Hadoop cluster as it contains updates to the Big Data SQL cells.

    Thanks!

    Marty

  • candan pehlivanoglu
    How should we gather external table stats on bda via bigdata...15.0
    Topic posted January 2, 2019 by candan pehlivanogluGreen Ribbon: 100+ Points, tagged Analytics, Big Data Appliance, Big Data SQL, Data Management, Hadoop in Data Management > Big Data public
    Title:
    How should we gather external table stats on bda via bigdata sql.
    Summary:
    Gather stats for external tables.
    Content:

    Hello all;

    We use bds for our external tables. We have daily jobs  to ingest these tables on big data and nealry 200 tables. These tables are partitioned on hive. But, on oracle site, since there isn't any partition column, we should gather all statistic every day. It is taking too much time. Do you have any other method for this operation. Since we use these tables on oracle, Oracle should be able to know statistic information.

     

    thank you.

     

  • candan pehlivanoglu
    BDA smart scan problem65.0
    Topic posted December 6, 2018 by candan pehlivanogluGreen Ribbon: 100+ Points, tagged Big Data Appliance, Big Data SQL in Data Management > Big Data public
    Title:
    BDA smart scan problem
    Summary:
    For one table offload is increasing
    Content:

    Hello all;

    We have a trouble about big data sql . Normally, our smart scan is working properly. But, for one table I saw predicate offload retries value is increasing. I don't know what I should check for one table. Could you help me?

    Version:
    3.1
  • Marty Gubar
    Big Data SQL 3.2 - Cloudera Certified5.0
    Topic posted October 29, 2018 by Marty GubarBlue Ribbon: 750+ Points, tagged Analytics, Big Data Appliance, Big Data SQL, Data Management, Hadoop, SQL, Tip in Data Management > Big Data public
    Title:
    Big Data SQL 3.2 - Cloudera Certified
    Summary:
    Big Data SQL 3.2.1 is now officially certified with Cloudera 5.14.1
    Content:

    Big Data SQL 3.2.1 is now officially Cloudera Certified (see Oracle certified products here).  The certification took place with Cloudera 5.14.1.  The testing captures real world functional integration usage of the product on a cluster that meets certain security requirements, including:

    • Kerberos
    • Sentry
    • HDFS Encryption
    • Auditing
    • Network Encryption. 

    It also examines the product architecture, including integration with Cloudera Manager and key CDH Components.  Finally, it reviews how the product leverages scale-out processing on Hadoop clusters.

  • Hany Ezzat Hassan
    Oracle Big Data Cloud services for all - Insights built to...24.8
    Topic posted July 25, 2018 by Hany Ezzat HassanSilver Medal: 2,000+ Points, tagged Big Data Appliance, Big Data SQL, Data Management, Tip in Data Management > Big Data public
    Title:
    Oracle Big Data Cloud services for all - Insights built to last
    Summary:
    Oracle BDCS
    Content:

    Big data remain a topic everyone seems to be talking about it, but still many wonder "what is big data really?"
    How is it changing the way researchers at companies, non-profits, governments, institutions, and other organizations are learning about the world around them.? Which technology provider should i trust ? What kind of data-set should be collected ? and many more similar questions we will try to answer during this post.

    Big Data came to existence as a way to handle  extremely large data sets that may be analyzed computationally to reveal patterns, trends, and associations, especially relating to human behavior and interactions. The more the data we could collect, the more complicated managing them became, thus different ways of handling and making sense of them had to be created. But most importantly, the more the data we could collect, the more information and insight we could get.

    For Oracle, Big data describes a holistic information management strategy that includes and integrates many new types of data and data management alongside traditional data.

    So, How is Big data used in practice ? 

    Big data analytics helps organizations tie up their data together and use it to identify new Business opportunities that leads to smarter business moves more efficient operation , higher profit and customer satisfaction  .

    • Targeting Customers

    Big data is used to better understand customers and their behaviors and preferences. Companies are keen to expand their traditional data sets with social media data, browser logs as well as text analytics and sensor data to get a more complete picture of their customers.

    • Humanizing Cities and Countries .

    Big Data allows cities to optimize traffic flows based on real time traffic information as well as social media and weather data.

    • Optimize Business processes.

    Big data is also increasingly used to optimize business processes. Retailers are able to optimize their stock based on predictions generated from social media data, web search trends and weather forecasts.

    • Personal qualification and performance optimization.

    Big data is not just for companies and governments but also for all of us individually. We can now benefit from the data generated from wearable devices such as smart watches or smart bracelets. Take the Up band from Jawbone as an example: the armband collects data on our calorie consumption, activity levels, and our sleep patterns. While it gives individuals rich insights, the real value is in analyzing the collective data.

    • Financial trading.

    High-Frequency Trading (HFT) is an area where big data finds a lot of use today. Here, big data algorithms are used to make trading decisions. Today, the majority of equity trading now takes place via data algorithms that increasingly take into account signals from social media networks and news websites to make, buy and sell decisions in split seconds.

    Why Oracle Big data  ?

    In a natural evolution from database and data warehouses, where Oracle is the #1 enterprise choice, Oracle invests in Big Data allowing you to gain a competitive advantage with an information strategy and big data architecture focused on data liquidity. Oracle extends your data management abilities to getting the data you want into the shape you need with the minimum time and effort.

    1. Oracle has got the biggest number of successful implementations a cross business industries .
    2. Biggest batch of tools and facilities makes easier and faster development .
    3. Biggest scalability and speed for data processing .
    4. Affordable price compared to other vendor experimental products .
    5. An engineered system called Big Data Appliance .

    Oracle Big Data Appliance is a flexible, high-performance, secure platform for running diverse workloads on Hadoop and NoSQL systems. With Oracle Big Data SQL, Oracle Big Data Appliance extends Oracle’s industry-leading implementation of SQL to Hadoop and NoSQL systems. By combining the newest technologies from the Hadoop ecosystem and powerful Oracle SQL capabilities together on a single pre-configured platform, Oracle Big Data Appliance is uniquely able to support rapid development of new Big Data applications and tight integration with existing relational data.

    And then comes Big Data in the Cloud @cloud.oracle.com

    Using the power of the cloud, Oracle Big Data becomes accessible to all, helping you get the most out of big data technologies today, and in minutes.

    1. Cost reduction. Big data technologies such as Hadoop and cloud-based analytics bring significant cost advantages when it comes to storing large amounts of data – plus they can identify more efficient ways of doing business.
    2. Big Data Connectors. Integrated with cloud and on-premises services. You can seamlessly integrate with other Oracle services using big data connectors.
    3. Elastic. Start small and grow your cluster based on processing needs. Scale to handle petabytes of data, no additional setup & administrators fees.
    4. Faster, better decision making. With the speed of Hadoop and in-memory analytics, combined with the ability to analyze new sources of data, businesses are able to analyze information immediately – and make decisions based on what they’ve learned.
    5. Oracle managed  service . Have Oracle manage the cluster for you, or manage it yourself with easy-to-use tooling. Single-click patching and upgrades.

    Together Oracle Big Data Cloud Service and Oracle Big Data SQL Cloud Service are dedicated, elastic, secure and comprehensive, delivering the power of Hadoop and Spark, which can be fully integrated with existing enterprise data in Oracle Database and Oracle Applications.

    With an additional choice of Oracle’s Big Data Cloud Service – Compute Edition we now deliver a Big Data Platform designed for the enterprises using open source technologies like Apache Spark and Apache Hadoop. In this service we utilize Oracle’s Infrastructure Cloud Services to deliver a managed, secure, elastic and integrated Platform for all Big Data Workloads.

    Drive innovation and business transformation using the industry’s #1 Enterprise Cloud Platform, the way you like.

  • Jean-Pierre Dijcks
    Welcome to the Big Data Forum25.0
    Topic posted July 19, 2018 by Jean-Pierre DijcksSilver Medal: 2,000+ Points, tagged Big Data Appliance, Big Data SQL, Data Management, Hadoop, Python, Spark, Spatial and Graph, SQL, Tip, Zeppelin in Data Management > Big Data public
    Title:
    Welcome to the Big Data Forum
    Summary:
    Welcome message for the big data cloud forum
    Content:
    Welcome everyone to the new Big Data Cloud Service (BDCS) Community.

    Here you'll be able to ask any questions you have about big data. Ranging from the BDCS service itself, to its components including Big Data Manager, Cloudera CDH, Data Science leveraging ORAAH, Spark, Python, our Zeppelin notebook support and Spatial and Graph, as well as the SQL engines like Big Data SQL, Impala, Hive and more.

    Remember that this is a community site and the people helping here are all part of a community of big data enthusiasts - so please be respectful and helpful.

    In order to get the best answers as fast as possible it is important that you'll provide some basic information in your post.

    • Have a meaningful title to your post
    • If you got an error - tell us what the error specifically says and if you have logs or details, provide them
    • If you are trying to do something - it might help us if you tell us why your are trying to do this - sometimes there are other ways to achieve the same goal.

    Looking forward to having a great conversation and community.