Python datetime operations

Python has a module named datetime to work with date and time.

Below are few examples of various datetime operations in python.

import time

import datetime

print("Time in seconds: %s" %time.time())

print("Current date_time1: " , datetime.datetime.now()

print("Current date_time2: " ,datetime.datetime.now().strftime("%y-%m-%d-%H-%M"))

print("Current year: ", datetime.date.today().strftime("%Y"))

print("Current month: ", datetime.date.today().strftime("%B"))

print("Current week number: ", datetime.date.today().strftime("%W"))

print("Weekday of the week: ", datetime.date.today().strftime("%w"))

print("Day of year: ", datetime.date.today().strftime("%j"))

print("Day of the month: ", datetime.date.today().strftime("%d"))

print("Day of week: ", datetime.date.today().strftime("%A"))

Output:

Time in seconds: 1553207598.3700132

Current date_time1:  2019-03-21 22:33:18.370013

Current date_time2:  19-03-21-22-33

Current year:  2019

Current Month:  March

Week number of the year:  11

Weekday of the week:  4

Day of year:  080

Day of the month:  21

Day of week:  Thursday

File operations in Python

The different file operation in Python are

• Open a file
• Read or write
• Close the file

Open()

Python has a built-in function open() to open a file. The open() function is used to open files in our system, the filename is the name of the file to be opened.

f = open("<Filename>")

We can specify the mode while opening a file. In mode, we specify whether we want to read 'r', write 'w' or append 'a' to the file.

The default is reading in text mode. In this mode, we get strings when reading from the file.

Example

filename = "hello.txt"

file = open(filename, "r")

for line in file:

   print line,

Read ()

The read functions contains following methods

• read()           #return one big string
• readline        #return one line at a time
• readlines       #returns a list of lines

Write ()

This method writes a sequence of strings to the file.

• write ()          #Used to write a fixed sequence of characters to a file
• writelines()    #writelines can write a list of strings.

Append ()

The append function is used to append to the file instead of overwriting it.

To append to an existing file, simply open the file in append mode ("a")

Close()

When you’re done with a file, use close() to close it and free up any system

resources taken up by the open file

File Handling Examples

To open a text file, use:

f = open("hello.txt", "r")

To read a text file, use:

f = open("hello.txt","r")

print(f.read())

To read one line at a time, use:

f = open("hello".txt", "r")

print(f.readline())

To read a list of lines use:

f = open("hello.txt.", "r")

print(f.readlines())

To write to a file, use:

f = open("hello.txt","w")

write("Hello World")

f.close()

To write to a file, use:

f = open("hello.txt", "w")

text_lines = ["text line 1", "text line 2", "text line 3"]

f.writelines(text_lines)

f.close()

To append to file, use:

f = open("hello.txt", "a")

write("Hello World again")

f.close()

To close a file, use

f = open("hello.txt", "r")

print(f.read())

f.close()

Python Basics for beginners

Python is a general-purpose interpreted, interactive, object-oriented, and high-level programming language. It was created by Guido van Rossum during 1985- 1990.

There are two major Python versions, Python 2 and Python 3. Python 2 and 3 are quite different. This tutorial uses Python 3, because it more semantically correct and supports newer features.

First Python Program

The simplest command in python is the "print" command. The print command prints the output.

In Python 2, the "print" statement is not a function, and therefore it is invoked without parentheses. 

However, in Python 3, it is a function, and must be invoked with parentheses. 

Below is a simple syntax to print "Hello World!"

print("Hello World!")

Lines and Indentation

Python provides no braces to indicate blocks of code for class and function definitions or flow control. Blocks of code are denoted by line indentation, which is rigidly enforced.

The number of spaces in the indentation is variable, but all statements within the block must be indented the same amount. For example −

if True:

   print "True"

else:

   print "False"

However, the following block generates an error −

if True:

print "Answer"

print "True"

else:

print "Answer"

print "False"

Thus, in Python all the continuous lines indented with same number of spaces would form a block.

 Standard Data Types

Python has various standard data types that are used to define the operations possible on them and the storage method for each of them.

Python has five standard data types −

• Numbers
• String
• List
• Tuple
• Dictionary

Basic Operators

Operators are the constructs which can manipulate the value of operands.

Consider the expression 4 + 5 = 9. Here, 4 and 5 are called operands and + is called operator.

Types of Operator

Python language supports the following types of operators.

• Arithmetic Operators
• Comparison (Relational) Operators
• Assignment Operators
• Logical Operators
• Bitwise Operators
• Membership Operators
• Identity Operators

Functions

A function is a block of organized, reusable code that is used to perform a single, related action. Functions provide better modularity for your application and a high degree of code reusing.

Defining a Function

• You can define functions to provide the required functionality. Here are simple rules to define a function in Python.
• Function blocks begin with the keyword def followed by the function name and parentheses ( ( ) ).
• Any input parameters or arguments should be placed within these parentheses. You can also define parameters inside these parentheses.
• The first statement of a function can be an optional statement - the documentation string of the function or docstring.
• The code block within every function starts with a colon (:) and is indented.
• The statement return [expression] exits a function, optionally passing back an expression to the caller. A return statement with no arguments is the same as return None.

Syntax

def functionname( parameters ):

   "function_docstring"

   function_suite

   return [expression]

Machine Learning

What is Machine Learning (ML)?

Machine Learning is programming machines to make it capable of taking decision itself based on their experience.

Machine Learning is a system that can learn from example through self-improvement and without being explicitly coded by programmer. The breakthrough comes with the idea that a machine can singularly learn from the data (i.e., example) to produce accurate results.

Machine learning combines data with statistical tools to predict an output. This output is then used by corporate to make actionable insights. Machine learning is closely related to data mining and Bayesian predictive modelling. The machine receives data as input, use an algorithm to formulate answers.

Machine Learning v/s Traditional programming

Traditional programming differs significantly from machine learning.

In traditional programming, data and program are run on the computer to produce the output.

In Machine Learning, data and output are run on the computer to create a program. This program can be used in traditional programming.

Types of Learning

Learning is the process of converting experience into expertise or knowledge.

Learning can be broadly classified into categories, as mentioned below, based on the nature of the learning data and interaction between the learner and the environment.

•  Supervised learning:

o   The program is “trained” on a pre-defined set of “training examples”, which then facilitate its ability to reach an accurate conclusion when given new data.

o   Supervised learning can be further classified into two types:

Ø  Regression

Ø  Classification

o   Supervised learning is commonly used in real world applications, such as face and speech recognition, products or movie recommendations, and sales forecasting.

• Unsupervised learning:

o   The program is given a bunch of data and must find patterns and relationships therein.

o   It is the opposite of supervised learning.

o   Unsupervised learning is used to detect anomalies, outliers, such as fraud or defective equipment, or to group customers with similar behaviours for a sales campaign.

•  Semi-supervised Learning:

o   If some learning samples are labelled, but some other are not labelled, then it is semi-supervised learning.

o   It makes use of a large amount of unlabelled data for training and a small amount of labelled data for testing.

o   Semi-supervised learning is applied in cases where it is expensive to acquire a fully labelled dataset while more practical to label a small subset.

•  Reinforcement Learning:

o   Learning data gives feedback so that the system adjusts to dynamic conditions to achieve a certain objective.

o   The system evaluates its performance based on the feedback responses and reacts accordingly.

o   AI types like it, it is the most ambitious type of learning.

Similarly, there are four categories of machine learning algorithms as shown below −

• Supervised learning algorithm
•  Unsupervised learning algorithm
•  Semi-supervised learning algorithm
•  Reinforcement learning algorithm

However, the most commonly used ones are supervised and unsupervised learning.

Machine Learning Process

Purpose of Machine Learning

Machine learning can be seen as a branch of Artificial Intelligence (AI), since, the ability to change experience into expertise or to detect patterns in complex data is a mark of human or animal intelligence.

As a field of science, machine learning shares common concepts with other disciplines such as statistics, information theory, game theory, and optimization.

As a subfield of information technology, its objective is to program machines so that they will learn.

However, it is to be seen that, the purpose of machine learning is not building an automated duplication of intelligent behaviour but using the power of computers to complement and supplement human intelligence. For example, machine learning programs can scan and process huge databases detecting patterns that are beyond the scope of human perception.

Applications

Below are some of the applications of Machine Learning:

• Web search: ranking page based on what you are most likely to click on.
•  Finance: decide who to send what credit card offers to. Evaluation of risk on credit offers. How to decide where to invest money.
•  E-commerce:  Predicting customer churn. Whether or not a transaction is fraudulent.
•  Space exploration: space probes and radio astronomy.
•  Robotics: how to handle uncertainty in new environments. Autonomous. Self-driving car.
•  Social networks: Data on relationships and preferences. Machine learning to extract value from data.
•  Debugging: Use in computer science problems like debugging. Labour intensive process. Could suggest where the bug could be.

Challenges

The primary challenge of machine learning is the lack of data or the diversity in the dataset. A machine cannot learn if there is no data available. Besides, a dataset with a lack of diversity gives the machine a hard time. A machine needs to have heterogeneity to learn meaningful insight. It is rare that an algorithm can extract information when there are no or few variations. It is recommended to have at least 20 observations per group to help the machine learn. This constraint leads to poor evaluation and prediction.

Conclusion

Machine learning is used by many known organizations in many different fields. However, this technology is quite new and not very mature, so it contains many security and ethical problems. However, this technology can help us in dealing with a lot of problems such as security. Machine learning has a lot of scope and with advancement in technology, it will get better n better.

Basics of Data Analytics

 Data vs Information

Data usually refers to raw data, or unprocessed data. It is the basic form of data, data that hasn’t been analysed or processed in any manner.

For example:        Jeetendra,Jaiswal,36,Gamdevi,Road,Bhan,dup,West,78,Mum

Information is data that has been interpreted so that it has meaning for the user. Once the data is analysed, it is considered as information.

For example:        Jeetendra Jaiswal

365, Gamdevi Road

Bhandup (West),

Mumbai-400078

What is Data Analytics?

Data Analytics is the science of using various techniques and processes for examining raw data for extracting information from them. Data is extracted and processed using various techniques based on organizational requirements with the purpose of finding patterns and drawing conclusions about that information.

Organizations collect and analyse both real time and historic data related to their customers.

Using data analytics, organizations can identify and optimize their operations, identify the needs of their customers and provide them the best possible service, proactively identify the future trend and respond to it leading to a gain in this competitive market over others.

All these can help the organizations to optimize costs and increase their revenues.

Types of data analytics applications

Data Analytics can be broadly classified under 2 categories

1)       Exploratory Data Analysis (EDA)

An approach to analysing data sets to summarize their main characteristics, often with visual methods. A statistical model can be used or not, but primarily EDA is for seeing what the data can tell us beyond the formal modelling or hypothesis testing task.

2)       Confirmatory Data Analysis (CDA)

An approach where you evaluate your evidence using traditional statistical tools such as                 significance, inference, and confidence.

Data analytics can also be classified as

         I.            Quantitative Data Analysis (QtDA)

An approach where numerical data is analysed with quantifiable variables that can be compared or measured statistically.

       II.            Qualitative Data Analysis (QlDA)

An approach where we understand the content of non-numerical data like text, images, audio and video, including common phrases, themes and points of view.

More advanced types of data analytics

A.      Data mining

Data Mining is the analysis of large quantities of data to extract previously unknown, interesting patterns of data, unusual data and the dependencies.

B.       Business Intelligence

Business Intelligence techniques and tools are for acquisition and transformation of large amounts of unstructured business data to help identify, develop and create new strategic business opportunities.

C.       Statistical Analysis

Statistics is the study of collection, analysis, interpretation, presentation, and organization of data. The 2 main types of Statistical Analysis are:

1)       Descriptive statistics - data from the entire population or a sample is summarized with numerical descriptors such as mean, frequency, percentage, etc.

2)       Inferential statistics − It uses patterns in the sample data to draw inferences about the represented population or accounting for randomness. These inferences can be −

·         answering yes/no questions about the data (hypothesis testing)

·         estimating numerical characteristics of the data (estimation)

·         describing associations within the data (correlation)

·         modelling relationships within the data (E.g. regression analysis)

D.      Predictive Analytics

Predictive Analytics use statistical models to analyse current and historical data for forecasting (predictions) about future or otherwise unknown events.

E.       Text Analytics

Text Analytics, also referred to as Text mining provides a means of analysing documents, emails and other text-based content.

Use of Data Analytics

Data analytics initiatives support a wide variety of business uses. Below are few examples

1)       Banks and credit card companies analyse withdrawal and spending patterns to prevent fraud and identity theft.

2)       E-commerce companies and marketing services providers do clickstream analysis to identify website visitors who are more likely to buy a product or service based on navigation and page-viewing patterns.

3)       Mobile network operators examine customer data to forecast churn so they can take steps to prevent defections to business rivals; to boost customer relationship management efforts, they and other companies also engage in CRM analytics to segment customers for marketing campaigns and equip call centre workers with up-to-date information about callers.

4)       Healthcare organizations mine patient data to evaluate the effectiveness of treatments for cancer and other diseases.

Fundamentals to Data Analytics

1)       Identifying the Data

It is important for a data analytics project to identify where the valuable information resides and map it based on the 4V framework, defined by Volume, Variety, Velocity and Veracity.

2)       Data Quality

Data quality is the core aspect in data analytics that decides its intended use in business operations and decision making. The correctness and consistency of the data demonstrates its quality and fitment for use.

3)       Business Objectives

Clarity in defining your goals and objectives are essential for achieving success through analytics. Analysts need to have in mind the big-picture while building the conceptual framework and process useful in data analytics.

4)       Data Availability & Access

Data availability and access is the fundamental requirement to data analytics. Authorized personnel should be able to access the internal organizational data, and, the information external to the organization must be collected from reliable resources.

5)       Insight

The success of a data analytics project depends on the quantifiable insight it generates. The derived system should be able to provide timely and accurate answer to the business questions. This presents a valuable actionable insight that marks a way to tread and adds up to the value chain.

6)       Data Visualization

For a meaningful insight, it is a must to present the information in an appealing and insightful manner to the intended audiences. The business story and the user story should be represented with advanced visualization techniques for better clarity, with scope for interactive exploration.

7)       Data Practices

The right data analysis framework, a standard architecture for data interoperability, and strict compliance to data security & privacy norms creates a trusted environment for data analysis. These enablers help organizations to undertake projects that assure high data security for a project’s success and stakeholders’ buy-in.

Challenges in Data Analytics

1)       Handling Enormous Data in Less Time

Handling the data of any business or industry is itself a significant challenge, but when it comes to handling enormous data, the task gets much more difficult. Critical business decisions should be taken effectively, but we need to have strong IT infrastructure which can read the data faster and delivering real-time insights. To overcome this challenge, you can use Apache Hadoop’s MapReduce that helps in splitting the data of the application in small fragments. This process makes the data measurable.

2)        Visual Representation of Data

Another important task is the visual representation of data. You need to represent the data in an easy format that makes it readable and understandable to the audience. Handling an unstructured data and then representing in a visually attractive manner could be a difficult task. To recover this issue, the data analyst can utilize different types of graphs or tables to represent the data.

3)        Application Should Be Scalable

The major factor to consider is the scalability factor of the of the applications. Several organizations are facing the same issue where the volume of data has been increasing each passing day. Due to the multiple layers between the database and front-end, the data traversal takes time. To overcome this issue, the organizations should take care of the application’s architecture and technology to reduce performance issues and enhance scalability.

Guidelines for effective use of Data Analytics

1)       Define the Questions

Your questions will define your work process. So, define your questions and ask measurable and clear questions. Define your problem clearly and design the question in such a way that it either qualify or disqualify potential solutions.

2)       Set Appropriate Measurement Priorities

This point covers two different scenarios, i.e. decide what to measure and how to measure. You need to think about these situations. Deciding on how to measure the data is important before the data collection phase as it also has its own set of questions.

3)        Collect Data

After defining the questions and setting up the measurement priorities, now you need to collect the data. Try to keep your collected data in an organized way.

4)        Analyse and Make Data Useful

Now is the time to analyse the data. You can manipulate the data in multiple ways by plotting and searching correlations or by building a pivot table. The pivot table will help in sorting and filtering data and calculate the maximum, minimum, mean and standard deviation of your data.

5)       Interpret Results

Data Analytics is incomplete without compelling visualization. This is the time to interpret your data. Interpreting the data will answer all the data-related questions.

Conclusion

Data analytics is helping businesses to consider hundreds of parameters to predict with reasonable accuracy what will happen. Both structured and unstructured data are growing exponentially. With proper use of technology, this huge data can be used to make more accurate decisions which can help the organizations improve its operations, reduce costs, improve sales, provide better service to customers and improve its efficiency.

Difference between a structure and a class

• Structures are value type and Classes are reference type.
• Structures can not have constructors or destructors. Classes can have both constructors and destructors.
• Structures do not support Inheritance, while Classes support Inheritance.
• Classes are usually used for large amounts of data, whereas structs are usually used for smaller amounts of data.
• A structure couldn't be null whereas a class can be null.
• A structure can't be abstract whereas a class can be abstract.

Difference between Dataset.clone and Dataset.copy in C#

Dataset.clone copies just the structure of dataset (including all the datatables, schemas, relations and constraints). It doesn't copy the data.

Dataset.copy copies both the dataset structure and data.