Ask Ghassem - Recent activity in General

Recategorized: Battery data projects

Thu, 03 Mar 2022 09:42:53 +0000

Where can I find projects related to battery data?

How can you build dynamic pricing model with data only from rigid pricing?

Fri, 21 Jan 2022 06:44:31 +0000

I want to build a dynamic pricing model which means if product is too expansive for a client and there is a risk that we might loose a client we lower the price for them but if client doesn't care that much about the price we might increase price a little.

All the articles I've seen describe some kind of A/B testing for the pricing and then create a model.

I want to build a model only on the existing rigid pricing data. So I have prices offered to customers and I know who bought the product and who went to other company.

How can I do the increasing price part?

Text Mining, Artificial Neural Networks, Speech Processing, Cloud Computing in DS? Essential for a good Data Scientist ?

Wed, 27 Oct 2021 19:15:16 +0000

Classification of data object might be incorrect

Mon, 25 Oct 2021 15:26:46 +0000

I am learning a new Salesforce product (Evergage) for the company I work for. In the program's documentation they have listed a set of data objects as an example. It appears to me that the classification might be incorrect. Their system makes a division between 'catalog objects' and 'profile objects' and the example they have given is a banking institution. They classified Customer Credit Card as a profile object and Credit Card Level as a catalog object. Seems to me that it should be the other way i.e Customer Credit Card = catalog object and Credit Card Level = profile object. Maybe I am not reading the context correctly?

here is a link to an image with the complete classification: https://drive.google.com/file/d/1nG4aX4Ty_NoHxm04AQo1Ow61m3MZ3pXm/view?usp=sharing

Can Data Science solve this problem?

Sun, 24 Oct 2021 15:43:11 +0000

So, I live in Brazil, and I have a task for college that I don't know what data science method to use, if at all, to solve it. My idea is the following: We Brazilians have Real (BRL) as currency, and we of course have the dollar quotation value to see "how many Reais a dollar is worth". What I wanted to do was to make a research and see whether the Country News have any influence over this price. So for example, if Bolsonaro, our president, says some dumb stuff, the dollar got up in price, and vice versa. What I wanted to do was collect all dollar values and variance over a set time interval, and try and get webscraping to get the news over some economy sites. Here's my question then: How can I correlate the news with the dollar variance over a set time? Can data science do that? How do I preprocess this, if at all? Do I need to use bag-of-words? At least I heard so... Please help and thank you for reading.

Searching for movie dataset containing movie synopses/plots?

Thu, 27 May 2021 09:57:31 +0000

Hello
To build a hybrid recommendation system, I used the movielens 1M dataset, for the collaborative filtering part. Now, I'm looking for a database/dataset that contains descriptions/summaries/details/synopses/plots of movies for the content-based recommendation.
Is there someone who could help me and tell me where I can find a such dataset?
thank you in advance.

My GloVe word embeddings contain sentiment?

Sun, 03 Jan 2021 14:09:37 +0000

I've been researching sentiment analysis with word embeddings. I read papers that state that word embeddings ignore sentiment information of the words in the text. One paper states that among the top 10 words that are semantically similar, around 30 percent of words have opposite polarity e.g. happy - sad.

So, I computed word embeddings on my dataset (Amazon reviews) with the GloVe algorithm in R. Then, I looked at the most similar words with cosine similarity and I found that actually every word is sentimentally similar. (E.g. beautiful - lovely - gorgeous - pretty - nice - love). Therefore, I was wondering how this is possible since I expected the opposite from reading several papers. What could be the reason for my findings?

Two of the many papers I read:

Yu, L. C., Wang, J., Lai, K. R. & Zhang, X. (2017). Refining Word Embeddings Using Intensity Scores for Sentiment Analysis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(3), 671-681.
Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T. & Qin, B. (2014). Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 1: Long Papers, 1555-1565

Answered: How to remove unwanted Jupyter notebook kernels?

Fri, 30 Oct 2020 17:18:25 +0000

You can see the list of installed kernels:

jupyter kernelspec list

And then remove those you want with follows (replace the name of the kernel you want to remove in following command)

jupyter kernelspec remove kernel

How can this data be structured for mongodb

Sun, 19 Jul 2020 19:08:50 +0000

https://prnt.sc/tkr2g7 Hello I have a PFE about determining risks of pedestrians, and I have to make a simulator to generate data with something related to this, this is my first time working on this. I would like to know, the structure of data, I will be working with mangodb, so I would love to see an example on JSON

Answered: What are the differences among Data Science, Artificial Intelligence and Machine Learning?

Thu, 05 Mar 2020 03:06:12 +0000

This whitepaper contains great insights on each. In a nutshell:

Artificial Intelligence (AI) is a “a huge set of tools for making computers behave intelligently” and in an automated fashion. This includes voice assistants, recommendation systems, and self-driving cars.

Machine Learning (ML) is the “field of study that gives computers the ability to learn without being explicitly programed.” The lion’s share of ML involves computers learning patterns from existing data and applying it to new data in the form of making predictions, such as predicting whether an email is spam or not, whether a customer will churn or not, and diagnosing a particular piece of medical imaging.

Data Science (DS) is about making discoveries and creating insights from data and communicating these insights and discoveries to non-technical stakeholders.

Answered: How to convert Jupyter Notebook or a webpage to PDF using Chrome?

Thu, 27 Feb 2020 18:41:45 +0000

Please take a look at short video below. However it is shown for Google Colab, it can be applied to other document types (Any webpage!)

https://www.youtube.com/watch?v=pNbPQmrfVFY

Answered: How to share a Jupyter Notebook document on Google Colab?

Thu, 27 Feb 2020 16:19:54 +0000

Please watch the following short video which explains sharing Jupyter Notebooks on Google Colab:

https://www.youtube.com/watch?v=oJwe1bxA4jQ

Answered: What are the most common data types in data science?

Wed, 19 Feb 2020 17:33:08 +0000

We can model object attributes in the form of two main forms of Categorical and Numerical. A more detailed explanation is available here.

https://i.imgur.com/JLbmW8m.jpg

Answered: How to install Matplotlib

Sat, 08 Feb 2020 06:46:05 +0000

The easiest way is using Anaconda which comes with it pre-installed. You can also use Google Colab.

Answered: ideas and opinion on what kind of analyses needs to be done

Thu, 01 Aug 2019 12:31:05 +0000

It is a broad question and could have many answers. You can ask many questions and try to answer them based on your main business problem: "the impact of all the products that were discontinued last year on the customers and sales". For example:

You can analyze a hypothesis that discontinuing a specific product was a good decision or a bad decision?
- You can check the sale increase in similar not-discontinued products of the store to investigate if customers who used to buy the discontinued one still looking for it or not. You can build the null hypothesis and alternative hypothesis to analyze the impact statistically.
- You can review customer calls/emails after discontinuing and analyzing the sentiment of the customer for this decision. It could be done using machine learning.

You have to generate questions and try to answer them using the data you have. It totally depends on how you define impact in your statement. After generating the questions, and finding answers, you can recommend some advice based on your results. For example, if you find out discontinuing the product was a bad decision, you can recommend to continue it again.

Answered: What are the most important Python libraries for data science?

Mon, 08 Jul 2019 04:43:54 +0000

Based on what is offered here, The most important libraries are:

Python Libraries for Data Collection
- Beautiful Soup
- Scrapy
- Selenium
Python Libraries for Data Cleaning and Manipulation
- Pandas
- PyOD
- NumPy
- Spacy
Python Libraries for Data Visualization
- Matplotlib
- Seaborn
- Bokeh
Python Libraries for Modeling
- Scikit-learn
- TensorFlow
- PyTorch
Python Libraries for Model Interpretability
- Lime
- H2O
Python Libraries for Audio Processing
- Librosa
- Madmom
- pyAudioAnalysis
Python Libraries for Image Processing
- OpenCV-Python
- Scikit-image
- Pillow
Python Libraries for Database
- Psycopg
- SQLAlchemy
Python Libraries for Deployment
- Flask

Answer selected: How to use LinkedIn Learning (formerly Lynda.com) for free?

Wed, 22 May 2019 06:59:13 +0000

If you have a university or college email address, the following photos show you how to sign in Lynda.com using that account.

Click on Sign in on the top menu to see the following page:
Click on Sign in with Lynda, and then write your email address of your school or college (for example *Emails are not allowed*)
It will redirect you to the portal of your organization and after that, you can create a free account on Lynda to use all the contents for free as long as you can sign in this way.
Some city libraries' LinkedIn Learning:
- City of Toronto
- City of Richmond Hill

Answered: How to reshape in pandas dataframe?

Sat, 06 Apr 2019 21:12:13 +0000

Did you try to use reshape(-1,1)?

a = np.array([[1, 2, 3, 4],
         [5, 6, 7, 8],
         [9, 10, 11, 12]])
print(a.shape)
# output: (3, 4)

print(a.reshape(-1,1))
'''
output:
array([[ 1],
   [ 2],
   [ 3],
   [ 4],
   [ 5],
   [ 6],
   [ 7],
   [ 8],
   [ 9],
   [10],
   [11],
   [12]])
'''

Answered: Is digital marketing and marketing internships worth it for a data science student?

Fri, 05 Apr 2019 00:38:05 +0000

Depending on what you want to do, and what kind of methodologies you use for analyzing and reporting, an internship in this area could be considered as data science experience.

Answered: How to open Jupyter notebook files on Windows or Mac without web browser?

Sat, 16 Mar 2019 23:38:07 +0000

If you want an executable application that opens Jupyter notebooks for viewing, printing and executing, nterac (https://nteract.io/desktop) is here to serve you. On the other hand, it means we can open Jupyter notebooks as standalone executables, instead of running inside the browser.

However, If you just need preview and do not need to execute, and you have a Mac, use this: https://github.com/tuxu/nbviewer-app

https://cloud.githubusercontent.com/assets/836375/18421299/d95ad398-783b-11e6-8b23-d54cf7caad1e.png

Passing variable length sentences to Tensorflow LSTM

Mon, 11 Feb 2019 05:06:27 +0000

I have a tensorflow LSTM model for predicting the sentiment. I build the model with the maximum sequence length 150. (Maximum number of words) While making predictions, i have written the code as below:

batchSize = 32
maxSeqLength = 150

def getSentenceMatrix(sentence):
    arr = np.zeros([batchSize, maxSeqLength])
    sentenceMatrix = np.zeros([batchSize,maxSeqLength], dtype='int32')
    cleanedSentence = cleanSentences(sentence)
    cleanedSentence = ' '.join(cleanedSentence.split()[:150])
    split = cleanedSentence.split()
    for indexCounter,word in enumerate(split):
        try:
            sentenceMatrix[0,indexCounter] = wordsList.index(word)
        except ValueError:
            sentenceMatrix[0,indexCounter] = 399999 #Vector for unkown words
    return sentenceMatrix

input_text = "example data"
inputMatrix = getSentenceMatrix(input_text)

In the code i'm truncating my input text to 150 words and ignoring remaining data.Due to this my predictions are wrong.

cleanedSentence = ' '.join(cleanedSentence.split()[:150])

I know that if we have lesser length than sequence length we can pad with zero's. What we need to do if we have more length. Can you suggest me the best way to do this. Thanks in advance.

Retagged: What are the general steps in data cleaning?

Wed, 31 Oct 2018 17:41:42 +0000

Answered: How do you copy a list without updating the original list?

Tue, 30 Oct 2018 11:17:02 +0000

The best way is to how b is copied below. As we can see, a is impacted by any changes we make on c but not b,

a = [1,2,3]
b=a[:]
c=a

b.append(99)
c.append(4)
print ("a=",a,"b=",b,"c=",c)

outputs

a= [1, 2, 3, 4] b= [1, 2, 3, 99] c= [1, 2, 3, 4]

Answered: How should I reduce file size of a word document when I attach high resolution pictures?

Fri, 12 Oct 2018 06:37:43 +0000

Please take a look at the following short video that shows you the procedure:

https://www.youtube.com/watch?v=QDojovVa5No&feature=youtu.be

How to use Ask Data Science! ?

Wed, 03 Oct 2018 07:01:30 +0000

This post is for providing the best practices for using Ask Data Science!

Answered: How to upload images on Ask Data Science! website?

Sun, 16 Sep 2018 21:05:47 +0000

Please create an account on https://imgur.com, and upload your images there, then attach the direct URL of your image to your post. Creating the account on imgur.com is optional, but I highly recommend it because it will let you upload images with better qualities if you are a registered member.

This video helps you learn how to find the Direct Link to the image after uploading.