Getting Big Data From Redshift

28 בAugust 2016

As part of a machine learning data collection task, I wanted to retrieve ~350 GB of data from Redshift into an AWS server in order to load the data into an IPython notebook and test some machine learning models. Get It All With Python (Attempt #1) Using psycopg2 library to work with Redshift, I tried to fetch the result ('fetchall') into the server, using a simple python Docker container. I initiated the script, only to find out that the server's RAM is bursting out, practically killing the docker container, and the hosting storage. It made sense that the server can’t cope...
tags:
no comments

AWS db parameters group

8 בAugust 2015

I'm working on a project using a mysql docker container. One of the scripts I run to init the db contains the following statement: This works on my local db. In production I work with an AWS RDS db. Running the same statement on that db results in: Although using a super user to log in, I still get this command preventing me from setting the db. After digging a bit i found that RDS does not allow you to directly change parameters but you must use the db parameters groups, were you can define global etc. parameters manually on the AWS console.
tags:
no comments

Bower package manager in 3 steps

17 בDecember 2014

Why do I need it Lets say I have a web app that depends on Angular. I need to download angular.js, angular.min.js, and maybe a gzip version. Also I might need to download a few dependencies such as angular-route and angular-ui, store them in my source control and make sure the versions match. Also, I use google-map for angular, which depends on a specific angular version, among jquery version and other. I need to download and store them all and to make sure the versions doesn't collide. Bower solves dependencies and downloads them for you. Just like nuget does. Why not...
tags: , ,
no comments

Angular: Why Should You Care About $destroy when $timeout

29 בOctober 2014

$timeout is a service that is used to invoke a function after a given amount of time. For example, invoking an alert("hello world") call after 1000 milliseconds would look as following: $timeout – when will it stop? But, have you ever consider what happens with the timeout call when you leave the page your in? Let's say the user clicks on a link and moves into a different ng-view that has a new controller. Will the $timeout call still be fire, calling your function unexpectedly even after the user has left the page? It seems the $timeout call will still be invoked, even...
tags: ,
no comments

Custom Web Api Json Converter For A Single Property

23 בSeptember 2014

The web is well documented regarding usage of a custom Json Converter. Basically the idea is to inherit from JsonConvert, explore the current serialized type and use your own custom behavior when serializing. The custom Json Convert can be used explicitly by calling Json.Convert or implicitly by registering in Application_Start. Explicit: Implicit: The implicit registration will serialize every object that contains the specified type (DateTime? in our example) using the MyJsonDateTimeConverter Converter. The explicit usage will only affect the current serialized value. The other case Both technics won't work in the following case; I need the converter to work only on a specified property, but I cannot...
tags: , , ,
no comments

Html In an Email – Embedded Buttons

21 בSeptember 2014

Embedding an html with images as hyper links in an Email. I expected Gmail \ outlook to have a built-in solution for it but apparently that is not the case. As every mail client has its own style of presenting the email content, expect some difficulties. One pain that I had is image spacing – making two image to have no space between them. Here are a few approaches I came up using; Embedding Html Outlook:  attach your html file and choose "Insert as text" instead of "Open". Gmail: I used a chrome extension Particular image areas as links For example, if you have...
tags: ,
no comments

Working around FacebookSdk.net Permissions BUG FacebookAuthorize attribute

14 בJune 2014

“Over the past several months Facebook made changes to their application development APIs that were incompatible with the MVC Facebook support.” says N. Taylor Mullen, a developer for the facebooksdk.net team in his latest blog post So it seems the latest sdk version (6.6.0.0) is incompatible with the current Facebook API hence some features won’t work. One basic feature mentioned in the tutorial for using the SDK is the FacebookAuthorize attribute which allows asking permissions for the Facebook app. Could not convert string to integer: installed. Path 'data.permission', line 1, position 34. It seems people are aware of the problem but still haven't...
tags: , , ,
4 comments

Be careful what you LINQ for: Mixing LINQ to SQL with LINQ to Entities

28 בMay 2014

An exception I encountered today, led me to pay more attention about the differences between LINQ to SQL and LINQ to Entities. Here is what I learned; Imagine your in a school, and the students council has announced the winners in the school's yearly programming competition. We have a bunch of students, and the winners names. Let's get all the winning student, that's easy: using (var db = new School()) { //Create students list var students = new List<Student> { new Student { ID =...
no comments

Facebook’s Hack Programming Language, Quick Introduction

19 בMay 2014

A few days ago, Facebook has announced the release of Hack, a PHP based programming language as an open source project. Let’s take a look into the new kid in the block. What will be hack’s market share? PHP is used in leading content management systems like Drupal and WordPress, Facebook's website is PHP based along with 240 million other websites (about 39% out of all) . It takes 3.3% of the programming language market, and much more relevant, 80% of server side programming language, giving Hack a large market \ user base potential. The Compiler Hack is a PHP based programming language...
tags: , ,
one comment