I was wondering recently if I could use Oracle R Enterprise (ORE) to query Big Data SQL tables (i.e. Oracle Database external tables based on HDFS or Hive data), since I have never seen such a combination mentioned in the relevant Oracle documentation and white papers. I am happy to announce that the answer is an unconditional yes. In this …
Nonlinear regression using Spark – Part 1: Nonlinear models
Regression constitutes a very important topic in supervised learning. Its goal is to predict the value of one or more continuous target variables (responses) given the value of a -dimensional vector of input variables (predictors). More specifically, given a training data set comprising of observations , where , together with corresponding target values , the goal is to predict the …
Caution when installing Oracle R Distribution in Oracle Linux using Yum
Last week we tried to install Oracle R Distribution (ORD) in Oracle Linux 7.1 using Yum, which is the installation method recommended by Oracle. After following closely the instructions provided in the documentation, instead of the Oracle R Distribution 3.2.0, we found ourselves with the latest (3.2.3) version of GNU R installed. What had happened is that in our /etc/yum.repos.d, …
Limitations of Spark MLlib linear algebra module
A couple of days ago I stumbled upon some unexpected behavior of Spark MLlib (v. 1.5.2), while trying some ultra-simple operations on vectors. Consider the following Pyspark snippet: Clearly, what happens is that the unary operator – (minus) for vectors fails, giving errors for expressions like -x and -y+x, although x-y behaves as expected. The result of the last operation, …
How NOT to perform feature selection!
Cross-validation (CV) is nowadays being widely used for model assessment in predictive analytics tasks; nevertheless, cases where it is incorrectly applied are not uncommon, especially when the predictive model building includes a feature selection stage. I was reminded of such a situation while reading this recent Revolution Analytics blog post, where CV is used to assess both the feature selection …
Oracle R Enterprise 1.4: ore.make.names does not work for Oracle DB connections
I have reported in the past about some unexpected behavior issues of Oracle R Enterprise 1.4 ore.make.names function; nevertheless, back then I had only tried it with Hive connections. I tried to use it today with an Oracle database connection, and it doesn’t seem to work. Here is a reproducible example in Oracle Big Data Lite VM 4.2.1, using the …
Manipulating Hive tables with Oracle R connectors for Hadoop
In this post, we’ll have a look at how easy it is to manipulate Hive tables using Oracle R connectors for Hadoop (ORCH, presently known as Oracle R Advanced Analytics for Hadoop – ORAAH). We will use the weblog data from Athens Datathon 2015, which we have already loaded in a Hive table named weblogs, as described in more detail …
Using Ansible to install WebLogic 12c R2 and Fussion Middleware
Before a couple of days Oracle release WebLogic 12c R2 (12.2.1). There are a lot of cool features like Java EE 7 support and Multitenancy Support for WebLogic domains. Installation of WebLogic server along with ADF runtime (Fusion Middleware Infrastructure) are not hard but requires a lot of parameters to be configured and a significant time when you need to …
Augmenting PCA functionality in Spark 1.5
Surprisingly enough, although the relatively new Spark ML library (not to be confused with Spark MLlib) includes a method for principal components analysis (PCA), there is no way to extract some very useful information regarding the PCA transformation, namely the resulting eigenvalues (check the Python API documentation); and, without the eigenvalues, one cannot compute the proportion of variance explained (PVE), …
Log files exploration with Oracle Big Data Discovery 1.1
In a previous post, we described how we performed exploratory data analysis (EDA) in real-world log files, as provided by Skroutz.gr, the leading online company in Greece for online price comparison, in the context of Athens Datathon 2015. In the present post we will have a look at the same job as performed with Oracle Big Data Discovery (v. 1.1), …