This notebook contains the steps and code to demonstrate support of Retrieval Augumented Generation in watsonx.ai. It introduces commands for data retrieval, knowledge base building & querying, and model testing.
Some familiarity with Python is helpful. This notebook uses Python 3.10.
Retrieval Augmented Generation (RAG) is a versatile pattern that can unlock a number of use cases requiring factual recall of information, such as querying a knowledge base in natural language.
In its simplest form, RAG requires 3 steps:
This notebook contains the following parts:
Before you use the sample code in this notebook, you must perform the following setup tasks:
!pip install "langchain>=0.0.345" | tail -n 1
!pip install elasticsearch | tail -n 1
!pip install sentence_transformers | tail -n 1
!pip install humanize | tail -n 1
!pip install pandas | tail -n 1
!pip install rouge_score | tail -n 1
!pip install nltk | tail -n 1
!pip install wget | tail -n 1
!pip install "pydantic==1.10.0" | tail -n 1
!pip install "ibm-watson-machine-learning>=1.0.327" | tail -n 1
import os, getpass
import pandas as pd
import humanize
import random
from typing import Optional, Any, Iterable, List
This cell defines the credentials required to work with watsonx API for Foundation Model inferencing.
Action: Provide the IBM Cloud user API key. For details, see documentation.
credentials = {
"url": "https://us-south.ml.cloud.ibm.com",
"apikey": getpass.getpass("Please enter your WML api key (hit enter): ")
}
The API requires project id that provides the context for the call. We will obtain the id from the project in which this notebook runs. Otherwise, please provide the project id.
Hint: You can find the project_id as follows. Open the prompt lab in watsonx.ai. At the very top of the UI, there will be Projects / <project name> /. Click on the <project name> link. Then get the project_id from Project's Manage tab (Project -> Manage -> General -> Details).
try:
project_id = os.environ["PROJECT_ID"]
except KeyError:
project_id = input("Please enter your project_id (hit enter): ")
Download the test dataset. This dataset is used to calculate the metrics score for selected model, defined prompts and parameters.
import wget
questions_test_filename = 'questions_test.csv'
questions_train_filename = 'questions_train.csv'
questions_test_url = 'https://raw.github.com/IBM/watson-machine-learning-samples/master/cloud/data/RAG/questions_test.csv'
questions_train_url = 'https://raw.github.com/IBM/watson-machine-learning-samples/master/cloud/data/RAG/questions_train.csv'
if not os.path.isfile(questions_test_filename):
wget.download(questions_test_url, out=questions_test_filename)
if not os.path.isfile(questions_train_filename):
wget.download(questions_train_url, out=questions_train_filename)
filename_test = './questions_test.csv'
filename_train = './questions_train.csv'
test_data = pd.read_csv(filename_test)
train_data = pd.read_csv(filename_train)
Inspect data sample
train_data.head()
| qid | question | answers | |
|---|---|---|---|
| 0 | 1961 | where does diffusion occur in the excretory sy... | diffusion |
| 1 | 7528 | when did the us join world war one | April 6 , 1917 |
| 2 | 8685 | who played wilma in the movie the flintstones | Elizabeth Perkins |
| 3 | 6716 | when was the office of the vice president created | 1787 |
| 4 | 2916 | where does carbon fixation occur in c4 plants | in the mesophyll cells |
The current state-of-the-art in RAG is to create dense vector representations of the knowledge base in order to calculate the semantic similarity to a given user query.
We can generate dense vector representations using embedding models. In this notebook, we use Sentence Transformers all-MiniLM-L6-v2 to embed both the knowledge base passages and user queries. all-MiniLM-L6-v2 is a performant open-source model that is small enough to run locally.
A vector database is optimized for dense vector indexing and retrieval. This notebook uses Elasticsearch, a distributed, RESTful search and analytics engine, capable of performing both vector and lexical search. It is built on top of the Apache Lucene library, which offers good speed and performance with all-MiniLM-L6-v2 embedding model.
The dataset we are using is already split into self-contained passages that can be ingested by Elasticsearch.
The size of each passage is limited by the embedding model's context window (which is 256 tokens for all-MiniLM-L6-v2).
Load set of documents used further to build knowledge base.
knowledge_base_dir = "./knowledge_base"
my_path = f"{os.getcwd()}/knowledge_base"
if not os.path.isdir(my_path):
os.makedirs(my_path)
documents_filename = 'knowledge_base/psgs.tsv'
documents_url = 'https://raw.github.com/IBM/watson-machine-learning-samples/master/cloud/data/RAG/psgs.tsv'
if not os.path.isfile(documents_filename):
wget.download(documents_url, out=documents_filename)
documents = pd.read_csv(f"{knowledge_base_dir}/psgs.tsv", sep='\t', header=0)
documents['indextext'] = documents['title'].astype(str) + "\n" + documents['text']
documents = documents[:1000]
Note that you can feed a custom embedding function to be used by Elasticsearch. The performance of Elasticsearch may differ depending on the embedding model used.
from langchain.embeddings import SentenceTransformerEmbeddings
from langchain.embeddings.base import Embeddings
emb_func = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
You need to specify model_id that will be used for inferencing:
from ibm_watson_machine_learning.foundation_models.utils.enums import ModelTypes
model_id = ModelTypes.FLAN_UL2
We need to provide a set of model parameters that will influence the result:
from ibm_watson_machine_learning.metanames import GenTextParamsMetaNames as GenParams
from ibm_watson_machine_learning.foundation_models.utils.enums import DecodingMethods
parameters = {
GenParams.DECODING_METHOD: DecodingMethods.GREEDY,
GenParams.MIN_NEW_TOKENS: 1,
GenParams.MAX_NEW_TOKENS: 50
}
Model class.¶from langchain.llms import WatsonxLLM
watsonx_granite = WatsonxLLM(
model_id=model_id.value,
url=credentials.get("url"),
apikey=credentials.get("apikey"),
project_id=project_id,
params=parameters
)
This notebook focuses on self-managed cluster using IBM Cloud® Databases for Elasticsearch.
The following cell retrieves the Elasticsearch users, password, host and port from the environment if available and prompts you otherwise.
try:
esuser = os.environ["ESUSER"]
except KeyError:
esuser = input("Please enter your Elasticsearch user name (hit enter): ")
try:
espassword = os.environ["ESPASSWORD"]
except KeyError:
espassword = getpass.getpass("Please enter your Elasticsearch password (hit enter): ")
try:
eshost = os.environ["ESHOST"]
except KeyError:
eshost = input("Please enter your Elasticsearch hostname (hit enter): ")
try:
esport = os.environ["ESPORT"]
except KeyError:
esport = input("Please enter your Elasticsearch port number (hit enter): ")
By default Elasticsearch will start with security features like authentication and TLS enabled. To connect to the Elasticsearch cluster you’ll need to configure the Python Elasticsearch client to use HTTPS with the generated CA certificate in order to make requests successfully. Details can be found here. In this notebook certificate fingerprints will be used for authentication.
Verifying HTTPS with certificate fingerprints (Python 3.10 or later) If you don’t have access to the generated CA file from Elasticsearch you can use the following script to output the root CA fingerprint of the Elasticsearch instance with openssl s_client (docs):
The following cell retrieves the fingerprint information using a shell command and stores it in variable ssl_assert_fingerprint.
es_ssl_fingerprint = !openssl s_client -connect $eshost:$esport -showcerts </dev/null 2>/dev/null | openssl x509 -fingerprint -sha256 -noout -in /dev/stdin
es_ssl_fingerprint = es_ssl_fingerprint[0].split("=")[1]
es_ssl_fingerprint
We first create a regular Elasticsearch Python client connection. Then we pass it into LangChain's ElasticsearchStore wrapper together with the WatsonX model based embedding function.
Consult the LangChain documentation. For more information about ElasticsearchStore connector.
from langchain.vectorstores.elasticsearch import ElasticsearchStore
from elasticsearch import Elasticsearch
es_connection = Elasticsearch([f"https://{esuser}:{espassword}@{eshost}:{esport}"],
basic_auth=(esuser, espassword),
request_timeout=None,
ssl_assert_fingerprint=es_ssl_fingerprint)
knowledge_base = ElasticsearchStore(es_connection=es_connection,
index_name="test_index",
embedding=emb_func,
strategy=ElasticsearchStore.ApproxRetrievalStrategy(),
distance_strategy="DOT_PRODUCT")
Note: Could take several minutes if you don't have pre-built indices
if es_connection.indices.exists(index="test_index"):
es_connection.indices.delete(index="test_index")
_ = knowledge_base.add_texts(texts=documents.indextext.tolist(),
metadatas=[{'title': title, 'id': doc_id}
for (title, doc_id) in
zip(documents.title, documents.id)], # filter on these!
index_name="test_index",
ids=[str(i) for i in documents.id] # unique for each doc
)
Let's take a look in Elasticsearch what the LangChain wrapper has created. First we display the newly created index ("tables" in Elasticsearch are always called "index"). Note the field vector of type dense_vector with dot_product similarity.
dict(es_connection.indices.get(index="test_index"))
{'test_index': {'aliases': {},
'mappings': {'properties': {'metadata': {'properties': {'id': {'type': 'long'},
'title': {'type': 'text',
'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}}}},
'text': {'type': 'text',
'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}},
'vector': {'type': 'dense_vector',
'dims': 384,
'index': True,
'similarity': 'dot_product'}}},
'settings': {'index': {'routing': {'allocation': {'include': {'_tier_preference': 'data_content'}}},
'allocation': {'max_retries': '15'},
'number_of_shards': '1',
'provided_name': 'test_index',
'creation_date': '1701783528532',
'unassigned': {'node_left': {'delayed_timeout': '60m'}},
'number_of_replicas': '1',
'uuid': '3KTqTam9Q86jBfQhRfFEwg',
'version': {'created': '8070099'}}}}}
Verify the number of documents loaded into the Elasticsearch index.
doc_count = es_connection.count(index='test_index')["count"]
doc_count
1000
Let's retrieve a random document as a sample. Note the embedding in the vector field, that was generated with the WatsonX embedding model.
dict(es_connection.get(index="test_index", id=random.randint(0, len(documents)-1)))
{'_index': 'test_index',
'_id': '543',
'_version': 1,
'_seq_no': 542,
'_primary_term': 1,
'found': True,
'_source': {'text': "Disneyland Paris\nRobert A.M. Stern , Stanley Tigerman , and Robert Venturi ) decided on an exclusively American theme in which each hotel would depict a region of the United States . At the time of the opening in April 1992 , seven hotels collectively housing 5,800 rooms had been built . An entertainment , shopping , and dining complex based on Walt Disney World 's Downtown Disney was designed by Frank Gehry . With its towers of oxidised silver and bronze - coloured stainless steel under a canopy of lights , it opened as Festival Disney . For a projected daily attendance of 55,000 , Euro Disney planned to serve an estimated 14,000 people per hour inside the Euro Disneyland park . In order to accomplish this , 29 restaurants were built inside the park ( with a further 11 restaurants built at the Euro Disney resort hotels and five at Festival Disney ) . Menus and prices were varied with an American flavour predominant and Disney 's precedent of serving alcoholic beverages was continued in the park . 2,300 patio seats ( 30 % of park seating ) were installed to satisfy Europeans ' expected preference of eating outdoors in good weather . In test kitchens at Walt Disney World , recipes were adapted for European tastes . Walter Meyer , executive chef for menu development at Euro Disney and executive chef of food projects development at Walt Disney World noted , `` A few things we did need to",
'metadata': {'title': 'Disneyland Paris', 'id': 543},
'vector': [0.1608273833990097,
-0.00924146082252264,
0.0489940419793129,
0.00018571832333691418,
0.024950873106718063,
-0.07827921211719513,
-0.0023028047289699316,
-0.04363097995519638,
0.021570127457380295,
-0.0541875958442688,
-0.019806044176220894,
-0.05081046745181084,
-0.01326067466288805,
-0.020598646253347397,
0.08630602061748505,
-0.13281209766864777,
0.05466827377676964,
-0.026405736804008484,
-0.02779398299753666,
-0.050891976803541183,
0.0003286092833150178,
-0.12299656122922897,
0.020582394674420357,
0.036336373537778854,
-0.12116983532905579,
0.030568795278668404,
-0.003829085733741522,
-0.0056760297156870365,
-0.022274740040302277,
0.03437335416674614,
0.008353734388947487,
-0.023000819608569145,
-0.006197935435920954,
-0.03046833910048008,
0.05060919374227524,
-0.01695890538394451,
-0.033901624381542206,
-0.058880068361759186,
-0.07964247465133667,
-0.008515235036611557,
0.004373209550976753,
-0.006235950160771608,
0.0017159924609586596,
-0.018781445920467377,
-0.0247736144810915,
0.04589447006583214,
-0.07301649451255798,
-0.054529935121536255,
-0.025599239394068718,
0.06984278559684753,
0.0865006223320961,
0.17860035598278046,
0.011139917187392712,
-0.1331275999546051,
-0.02604636736214161,
0.032347213476896286,
0.018705694004893303,
-0.03845629096031189,
-0.0025478252209722996,
0.059136152267456055,
-0.011258444748818874,
0.0003150050179101527,
0.019544417038559914,
-0.011595631949603558,
-0.020715411752462387,
0.0008456922951154411,
-0.09581168740987778,
0.011563973501324654,
-0.07322415709495544,
-0.06985786557197571,
-0.01325069461017847,
-0.03735050931572914,
0.032335732132196426,
-0.005694623105227947,
0.03690445423126221,
0.009061780758202076,
-0.011704922653734684,
-0.031278543174266815,
-0.015579852275550365,
0.021211521700024605,
0.05193692445755005,
0.013349531218409538,
0.06283838301897049,
-0.03714454546570778,
0.0656796246767044,
-0.014140775427222252,
0.02122608944773674,
-0.015583635307848454,
-0.019356975331902504,
0.049856036901474,
0.00036351667949929833,
-0.047850582748651505,
-0.09029782563447952,
0.02178555354475975,
0.03610138222575188,
0.09123965352773666,
-0.0010501481592655182,
-0.016001420095562935,
0.025257155299186707,
-0.03475241735577583,
-0.02548142522573471,
0.07198473811149597,
0.08943480998277664,
-0.053371675312519073,
-0.02545582875609398,
-0.027844268828630447,
0.037875931710004807,
0.08111871778964996,
-0.00777529738843441,
0.004704047925770283,
-0.06893131881952286,
-0.02669345960021019,
0.08828988671302795,
-0.06711957603693008,
0.0068211629986763,
0.013651583343744278,
0.04369682818651199,
-0.11146480590105057,
0.11775282770395279,
0.03104480169713497,
0.06441424787044525,
0.05235312506556511,
0.07702932506799698,
0.07262061536312103,
-0.057454369962215424,
0.017625654116272926,
0.05675584450364113,
-6.138295124282191e-34,
-0.05514214187860489,
-0.013970271684229374,
0.011685529723763466,
-0.016882386058568954,
0.045198746025562286,
0.03178432956337929,
-0.0434182770550251,
0.016596706584095955,
0.017049593850970268,
-0.002288605784997344,
0.03868666663765907,
-0.05577396973967552,
-0.03188858553767204,
0.06354265660047531,
0.011101052165031433,
0.025199852883815765,
0.08074025064706802,
0.025598250329494476,
-0.034649383276700974,
0.005568527616560459,
0.0419381745159626,
-0.027045588940382004,
0.061924807727336884,
0.05689835920929909,
-0.015319138765335083,
0.05836136266589165,
-0.10142126679420471,
0.030496152117848396,
-0.01397608406841755,
0.018774569034576416,
-0.015128137543797493,
-0.08397293090820312,
-0.017449848353862762,
0.0425884947180748,
-0.027760880067944527,
-0.014734381809830666,
0.05978437140583992,
-0.015413075685501099,
-0.05181790143251419,
-0.04969605803489685,
-0.050859007984399796,
0.025967082008719444,
-0.02178897149860859,
0.02280515804886818,
-0.05652172490954399,
0.08203333616256714,
0.029196713119745255,
0.01957559958100319,
0.008157537318766117,
0.07040318846702576,
-0.08829785138368607,
-0.06014062464237213,
0.028116660192608833,
-0.04682013392448425,
-0.011724483221769333,
-0.00036111328518018126,
0.07343525439500809,
-0.03731647878885269,
0.05186592787504196,
-0.026546845212578773,
-0.0578148290514946,
0.1639488935470581,
-0.04144462198019028,
0.011172957718372345,
0.020319951698184013,
0.03337467089295387,
0.010542144998908043,
-0.005751743447035551,
-0.0018052591476589441,
-0.049541499465703964,
-0.02331126295030117,
-0.014088536612689495,
0.08554594218730927,
-0.042692288756370544,
0.026660433039069176,
0.012310405261814594,
0.04285228252410889,
0.0393347293138504,
0.019434403628110886,
0.08779621124267578,
0.029042750597000122,
0.037825047969818115,
0.04361192137002945,
0.02924904227256775,
0.010273465886712074,
-0.0099522415548563,
-0.04601975530385971,
0.06438444554805756,
0.0263894684612751,
-0.003996293991804123,
0.021126272156834602,
-0.0496196448802948,
0.06351929157972336,
0.023018047213554382,
-0.06379528343677521,
-2.763807072961e-33,
-0.016615359112620354,
0.03713582083582878,
0.03941871225833893,
-0.06166594848036766,
0.03897788003087044,
0.009257584810256958,
-0.08381671458482742,
-0.05325877666473389,
-0.0009747838485054672,
-0.08261662721633911,
-0.07321371138095856,
-0.0015724319964647293,
0.020787730813026428,
-0.016585741192102432,
-0.03737887740135193,
0.016179224476218224,
0.0265456922352314,
0.06895515322685242,
0.0192469023168087,
0.07708780467510223,
0.0007697084220126271,
0.061545614153146744,
-0.03579824045300484,
0.0645710825920105,
-0.04512811079621315,
0.054018229246139526,
0.04292798787355423,
-0.028586454689502716,
-0.07004166394472122,
-0.06181905418634415,
-0.07102201879024506,
0.08334315568208694,
0.05855872854590416,
0.030540063977241516,
0.05632516369223595,
-0.07653852552175522,
-0.07615432888269424,
0.05943736433982849,
-0.06764736026525497,
-0.017196033149957657,
0.013732693158090115,
-0.045250896364450455,
-0.031714364886283875,
0.013478787615895271,
0.07897596806287766,
0.04424726590514183,
-0.05394540727138519,
-0.17217980325222015,
-0.0034057265147566795,
0.0017981452401727438,
-0.054177626967430115,
-0.08411402255296707,
-0.10816234350204468,
-0.05358577147126198,
0.009894204325973988,
-0.011603876948356628,
-0.0018955256091430783,
0.0016424977220594883,
0.07428590953350067,
0.04523872211575508,
-0.016205301508307457,
0.04045949503779411,
0.08126100897789001,
0.06719667464494705,
-0.06584273278713226,
-0.08914053440093994,
-0.0014294369611889124,
-0.008543360978364944,
-0.017635677009820938,
-0.008460655808448792,
-0.01889103464782238,
0.03388270363211632,
0.021223241463303566,
0.07328897714614868,
-0.033659107983112335,
-0.002262151101604104,
0.07493246346712112,
0.014647167176008224,
0.012692990712821484,
0.01866152510046959,
-0.0344412736594677,
-0.018760044127702713,
0.005113556515425444,
-0.06728928536176682,
-0.01274380274116993,
0.007995426654815674,
-0.03184118494391441,
-0.013801141642034054,
-0.05083214491605759,
-0.0041078608483076096,
-0.009134260006248951,
0.02733713947236538,
-0.029491441324353218,
0.015812458470463753,
0.04389653354883194,
-5.380731593618293e-08,
-0.01320723071694374,
0.040376149117946625,
-0.002480017952620983,
0.015004066750407219,
-0.05668551102280617,
-0.14270390570163727,
0.05389583855867386,
0.0024976234417408705,
-0.021888799965381622,
0.004126595798879862,
0.03651357814669609,
0.052563153207302094,
0.013877415098249912,
0.017931966111063957,
-0.03830256313085556,
0.03171682357788086,
-0.026293877512216568,
-0.007108695339411497,
0.0082095330581069,
0.16406266391277313,
0.040270861238241196,
-0.0007145312847569585,
-0.003965656738728285,
-0.04688337445259094,
0.06287135928869247,
-0.05251444876194,
-0.03849373385310173,
0.03545830026268959,
-0.0041947560384869576,
-0.09576528519392014,
-0.011105438694357872,
-0.07131951302289963,
-0.08223553001880646,
0.03630261868238449,
-0.007466008421033621,
-0.04966267570853233,
-0.056808460503816605,
-0.07958333939313889,
-0.07775544375181198,
-0.026769516989588737,
-0.032380521297454834,
-0.07226215302944183,
-0.04178983345627785,
-0.0428248755633831,
0.056745972484350204,
0.06591080874204636,
-0.04346395283937454,
0.02911977283656597,
0.007528978865593672,
0.04358380287885666,
-0.08932475745677948,
0.05168084055185318,
-0.05855173245072365,
-0.059789709746837616,
0.053092364221811295,
-0.0476100891828537,
0.032110050320625305,
0.042803432792425156,
0.08270702511072159,
-0.0974821001291275,
0.011330148205161095,
-0.0013125179102644324,
-0.07645995914936066,
0.0038770961109548807]}}
Display the total size and indexing time of the new index in Elasticsearch.
index_stats = es_connection.indices.stats(index="test_index").get('_all').get('primaries')
print("Index size: " + humanize.naturalsize(index_stats.get('store').get('size_in_bytes')))
print("Indexing time: " + humanize.precisedelta(index_stats.get('indexing').get('index_time_in_millis')/1000, minimum_unit='minutes'))
Index size: 9.4 MB Indexing time: 0.02 minutes
RetrievalQA is a chain to do question answering.
Hint: To use Chain interface from LangChain with watsonx.ai models you must call model.to_langchain() method.
It returns WatsonxLLM wrapper compatible with LangChain CustomLLM specification.
The prompts we will use to test the RAG flow
questions_and_answers = {
'names of founding fathers of the united states?': "Thomas Jefferson::James Madison::John Jay::George Washington::John Adams::Benjamin Franklin::Alexander Hamilton",
'who played in the super bowl in 2013?': 'Baltimore Ravens::San Francisco 49ers',
'when did bucharest become the capital of romania?': '1862'
}
Fetch paragraphs similar to the question
from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(llm=watsonx_granite, chain_type="stuff", retriever=knowledge_base.as_retriever(), return_source_documents=True)
results = []
for question in questions_and_answers.keys():
result = qa({"query": question})
results.append(result)
Get the set of chunks for one of the questions.
for idx, result in enumerate(results):
print("=========")
print("Question = ", result['query'])
print("Answer = ", result['result'])
print("Expected Answer(s) (may not be appear with exact wording in the dataset) = ", questions_and_answers[result['query']])
print("\n")
print("Source documents:")
print(*(x.page_content for x in result['source_documents']), sep='\n')
print("\n")
========= Question = names of founding fathers of the united states? Answer = John Adams , Benjamin Franklin , Alexander Hamilton , John Jay , Thomas Jefferson , James Madison , and George Washington Expected Answer(s) (may not be appear with exact wording in the dataset) = Thomas Jefferson::James Madison::John Jay::George Washington::John Adams::Benjamin Franklin::Alexander Hamilton Source documents: Founding Fathers of the United States ^ Burstein , Andrew . `` Politics and Personalities : Garry Wills takes a new look at a forgotten founder , slavery and the shaping of America '' , Chicago Tribune ( November 09 , 2003 ) : `` Forgotten founders such as Pickering and Morris made as many waves as those whose faces stare out from our currency . '' ^ Jump up to : Rafael , Ray . The Complete Idiot 's Guide to the Founding Fathers : And the Birth of Our Nation ( Penguin , 2011 ) . Jump up ^ `` Founding Fathers : Virginia '' . FindLaw Constitutional Law Center . 2008 . Retrieved 2008 - 11 - 14 . Jump up ^ Schwartz , Laurens R. Jews and the American Revolution : Haym Solomon and Others , Jefferson , North Carolina : McFarland & Co. , 1987 . Jump up ^ Kendall , Joshua . The Forgotten Founding Father : Noah Webster 's Obsession and the Creation of an American Culture ( Penguin 2011 ) . Jump up ^ Wright , R.E. ( 1996 ) . `` Thomas Willing ( 1731 - 1821 ) : Philadelphia Financier and Forgotten Founding Father '' . Pennsylvania History . 63 ( 4 ) : 525 -- 560 . doi : 10.2307 / 27773931 ( inactive 2017 - 01 - 15 ) . JSTOR 27773931 . Jump up ^ `` A Patriot of Early New England '' , New York Times ( December 20 , 1931 ) Founding Fathers of the United States President of Congress Peyton Randolph New Hampshire John Sullivan Nathaniel Folsom Massachusetts Bay Thomas Cushing Samuel Adams John Adams Robert Treat Paine Rhode Island Stephen Hopkins Samuel Ward Connecticut Eliphalet Dyer Roger Sherman Silas Deane New York Isaac Low John Alsop John Jay James Duane Philip Livingston William Floyd Henry Wisner Simon Boerum New Jersey James Kinsey William Livingston Stephen Crane Richard Smith John De Hart Pennsylvania Joseph Galloway John Dickinson Charles Humphreys Thomas Mifflin Edward Biddle John Morton George Ross The Lower Counties Caesar Rodney Thomas McKean George Read Maryland Matthew Tilghman Thomas Johnson , Junr William Paca Samuel Chase Virginia Richard Henry Lee George Washington Patrick Henry , Junr Richard Bland Benjamin Harrison Edmund Pendleton North Carolina William Hooper Joseph Hewes Richard Caswell South Carolina Henry Middleton Thomas Lynch Christopher Gadsden John Rutledge Edward Rutledge See also Virginia Association First Continental Congress Carpenters ' Hall Declaration and Resolves of the First Continental Congress Retrieved from `` https://en.wikipedia.org/w/index.php?title=Founding_Fathers_of_the_United_States&oldid=815247535 '' Categories : Age of Enlightenment American Revolution Articles about multiple people Patriots in the American Revolution Political leaders of the American Revolution National founders Hidden categories : Webarchive template wayback links Pages with DOIs inactive since 2017 Use mdy dates from April 2011 Talk Read Contents About Wikipedia Wikiquote Bân - lâm - gú Башҡортса Bikol Central Български Brezhoneg Català Čeština Cymraeg Deutsch Ελληνικά Español Esperanto فارسی Français 한국어 Հայերեն हिन्दी Ido Bahasa Indonesia Interlingua Íslenska Italiano עברית Latina Lietuvių Magyar Bahasa Melayu Nederlands 日本 語 Norsk Polski Português Română Founding Fathers of the United States Founding Fathers of the United States - wikipedia Founding Fathers of the United States Jump to : navigation , search Declaration of Independence , a painting by John Trumbull depicting the Committee of Five presenting their draft to the Congress on June 28 , 1776 Signature page of Treaty of Paris ( 1783 ) ; the treaty was negotiated by John Adams , Benjamin Franklin and John Jay . The Founding Fathers of the United States were those individuals of the Thirteen Colonies in North America who led the American Revolution against the authority of the British Crown in word and deed and contributed to the establishment of the United States of America . Historian Richard B. Morris in 1973 identified the following seven figures as the key Founding Fathers : John Adams , Benjamin Franklin , Alexander Hamilton , John Jay , Thomas Jefferson , James Madison , and George Washington . Adams , Jefferson , and Franklin were members of the Committee of Five that drafted the Declaration of Independence . Hamilton , Madison , and Jay were authors of The Federalist Papers , advocating ratification of the Constitution . The constitutions drafted by Jay and Adams for their respective states of New York ( 1777 ) and Massachusetts ( 1780 ) were heavily relied upon when creating language for the US Constitution Jay , Adams and Franklin negotiated the Treaty of Paris ( 1783 ) that would end the American Revolutionary War . Washington was Commander - Founding Fathers of the United States Prior to , and during the 19th century , they were referred to as simply the `` Fathers '' . The term has been used to describe the founders and first settlers of the original royal colonies . Contents ( hide ) 1 Background 2 Interesting facts and commonalities 2.1 Education 2.1. 1 Colleges attended 2.1. 2 Advanced degrees and apprenticeships 2.1. 2.1 Doctors of medicine 2.1. 2.2 Theology 2.1. 2.3 Legal apprenticeships 2.1. 3 Self - taught or little formal education 2.2 Demographics 2.3 Political experience 2.4 Occupations and finances 2.5 Religion 2.6 Ownership of slaves and position on slavery 2.7 Attendance at conventions 2.8 Spouses and children 2.9 Charters of freedom and historical documents of the United States 2.10 Post-constitution life 2.11 Youth and longevity 2.12 Founders who were not signatories or delegates 3 Legacy 3.1 Institutions formed by Founders 3.2 Scholarship on the Founders 3.2. 1 Living historians whose focus is the Founding Fathers 3.2. 2 Noted collections of the Founding Fathers 3.3 In stage and film 3.4 Children 's books 4 See also 5 Notes 6 References 7 External links Background ( edit ) The Albany Congress of 1754 was a conference attended by seven colonies , which presaged later efforts at cooperation . The Stamp Act Congress of 1765 included representatives from nine colonies . The First Continental Congress met briefly in Philadelphia , Pennsylvania in 1774 , consisting of fifty - six delegates from twelve of the thirteen colonies ( excluding Georgia ) that ========= Question = who played in the super bowl in 2013? Answer = Baltimore Ravens Expected Answer(s) (may not be appear with exact wording in the dataset) = Baltimore Ravens::San Francisco 49ers Source documents: Super Bowl XLVII responded to the claim on Twitter in jest , tweeting `` There is no conspiracy . I pulled the plug . '' Box score ( edit ) Super Bowl XLVII : Baltimore Ravens 34 , San Francisco 49ers 31 Total Ravens ( AFC ) 7 14 7 6 34 49ers ( NFC ) 17 8 31 at Mercedes - Benz Superdome in New Orleans , Louisiana Date : February 3 , 2013 Game time : 5 : 31 p.m. CST Game weather : Played indoors ( dome stadium ) Game attendance : 71,024 Referee : Jerome Boger TV announcers ( CBS ) : Jim Nantz , Phil Simms , Steve Tasker , Solomon Wilcots Gamebook ( hide ) Scoring summary Quarter Time Drive Team Scoring information Score Plays Yards TOP BAL SF 10 : 36 6 51 2 : 29 BAL Anquan Boldin 13 - yard touchdown reception from Joe Flacco , Justin Tucker kick good 7 0 3 : 58 12 62 6 : 38 SF 36 - yard field goal by David Akers 7 7 : 10 10 75 4 : 43 BAL Dennis Pitta 1 - yard touchdown reception from Flacco , Tucker kick good 14 1 : 45 56 0 : 22 BAL Jacoby Jones 56 - yard touchdown reception from Flacco , Tucker kick good 21 0 : 00 8 71 1 : 45 SF 27 - yard field goal by Akers 21 6 14 : 49 -- -- -- BAL J. Jones 108 - Super Bowl XLVII for the 2012 season . The Ravens defeated the 49ers by the score of 34 -- 31 , handing the 49ers their first Super Bowl loss in franchise history . The game was played on February 3 , 2013 , at Mercedes - Benz Superdome in New Orleans , Louisiana . This was the tenth Super Bowl to be played in New Orleans , equaling Miami 's record of ten in an individual city . For the first time in Super Bowl history , the game featured two brothers coaching against each other -- Jim and John Harbaugh , head coaches of the San Francisco 49ers and Baltimore Ravens , respectively -- earning it the nickname Harbaugh Bowl . In addition , Super Bowl XLVII was the first to feature two teams that had undefeated records in previous Super Bowl games ( Baltimore , 1 -- 0 ; San Francisco , 5 -- 0 ) . The 49ers , who posted a regular - season record of 11 -- 4 -- 1 , entered the game seeking their sixth Super Bowl win in team history ( and first since Super Bowl XXIX at the end of the 1994 season ) , which would have tied the Pittsburgh Steelers for the most by a franchise . The Ravens , who posted a 10 -- 6 regular - season record , made their second Super Bowl appearance in 12 years , having previously won Super Bowl XXXV . Ray Lewis , the Super Bowl XLVII preview : A minute - by - minute breakdown '' . USA Today . Retrieved February 2 , 2013 . Jump up ^ `` Beyonce half time performance '' . NFL.Com . February 4 , 2013 . Retrieved February 5 , 2013 . Jump up ^ `` Beyonce wows at half - time show '' . BBC News . February 4 , 2013 . Retrieved February 5 , 2013 . Jump up ^ `` Beyonce Draws Estimated 104 Million to Super Bowl Halftime '' . ^ Jump up to : `` Play - by - Play '' . ESPN . February 3 , 2013 . Retrieved February 4 , 2013 . Jump up ^ Inman , Cam ( February 3 , 2013 ) . `` Super Bowl 2013 : San Francisco 49ers ' furious comeback falls short in 34 -- 31 loss to Baltimore Ravens -- San Jose Mercury News '' . San Jose Mercury News . Retrieved February 4 , 2013 . ^ Jump up to : `` Jacoby Jones returns kick 108 yards '' . ESPN.com . February 4 , 2013 . Retrieved April 24 , 2016 . Jump up ^ `` Watch Baltimore Ravens vs. Denver Broncos ( 01 / 12 / 2013 ) '' . NFL . Retrieved February 4 , 2013 . ^ Jump up to : Rosenthal , Gregg ( February 3 , 2013 ) . `` Jacoby Jones ' 108 - yard return TD a Super Bowl record '' . NFL . Retrieved Super Bowl XLVII MVP Award '' . Forbes.com . February 4 , 2013 . Retrieved February 6 , 2013 . Jump up ^ Heitner , Darren ( April 18 , 2013 ) . `` Is It Worth Spending $4 Million On A Super Bowl Commercial ? '' . Forbes . Retrieved February 4 , 2013 . Jump up ^ Collins , Scott ( February 5 , 2013 ) . `` Super Bowl ratings dip slightly from last year '' . Los Angeles Times . Retrieved February 5 , 2013 . ^ Jump up to : `` New Orleans to host 10th Super Bowl in 2013 '' . ESPN.com . Associated Press . May 19 , 2009 . Retrieved January 14 , 2011 . Jump up ^ Ally Burguieres Designs Official NFL Beads with Courtyard by Marriott for SuperBowl -- WGNO -- YouTube on YouTube Jump up ^ Wilner , Barry ( January 21 , 2013 ) . `` Ravens dominate Pats , set up ' Harbaugh Bowl ' '' . NBC Sports . Associated Press . Retrieved January 21 , 2013 . Jump up ^ Brinson , Will . `` Sorting the Sunday Pile , Divisional Round : Harbaugh Bowl still lives '' . CBSSports.com . CBS Interactive . Retrieved January 21 , 2013 . Jump up ^ Schwab , Frank . `` HarBowl ! Harbaugh brothers Jim and John to square off in Super Bowl '' . Yahoo ! Sports . Retrieved January 21 , 2003 . Jump up ^ `` The ========= Question = when did bucharest become the capital of romania? Answer = 1862 Expected Answer(s) (may not be appear with exact wording in the dataset) = 1862 Source documents: Bucharest EU ) Density 9,237 / km ( 23,920 / sq mi ) Metro 2,412,530 Demonym ( s ) Bucharester ( en ) bucureștean , bucureșteancă ( ro ) Time zone UTC + 2 ( EET ) Summer ( DST ) UTC + 3 ( EEST ) Postal code 0xxxxx Area code ( s ) + 40 x1 Car plate prefix GDP ( Nominal ) € 47 billion - Per capita PPP € 40,400 -- Per capita € 20,000 Website pmb.ro Romanian law stipulates that Bucharest has a special administrative status which is equal to that of a County ; Bucharest metropolitan area is a proposed project . Bucharest ( / ˈb ( j ) uː kərɛst / ; Romanian : București ( bukuˈreʃtj ) ( listen ) ) is the capital and largest city of Romania , as well as its cultural , industrial , and financial centre . It is located in the southeast of the country , at 44 ° 25 ′ 57 '' N 26 ° 06 ′ 14 '' E / 44.43250 ° N 26.10389 ° E / 44.43250 ; 26.10389 Coordinates : 44 ° 25 ′ 57 '' N 26 ° 06 ′ 14 '' E / 44.43250 ° N 26.10389 ° E / 44.43250 ; 26.10389 , on the banks of the Dâmbovița River , less than 60 km ( 37.3 mi ) north of the Danube River and the Bulgarian border . Bucharest was first mentioned in Bucharest Bucharest - wikipedia Bucharest This article contains too many pictures , charts or diagrams for its overall length . Please help to improve this article by removing or adjusting the sandwiching of text between two images and indiscriminate galleries in accordance with the Manual of Style on use of images . ( Learn how and when to remove this template message ) Capital city in None , Romania Bucharest București Capital city From top , left to right : Central University Library Palace of the Parliament Arcul de Triumf Romanian Athenaeum Modernist skyscrapers CEC Palace Royal Palace of Bucharest Flag Coat of arms Nickname ( s ) : Micul Paris ( The Little Paris ) , Paris of the East Motto ( s ) : Patria și dreptul meu ( The Homeland and my right ) Bucharest Location of Bucharest in Romania Show map of Romania Bucharest Bucharest ( Europe ) Show map of Europe Coordinates : 44 ° 25 ′ 57 '' N 26 ° 6 ′ 14 '' E / 44.43250 ° N 26.10389 ° E / 44.43250 ; 26.10389 Country Romania County None First attested 1459 Government Mayor Gabriela Firea ( PSD ) Prefect Speranţa Cliseru Area Capital city 228 km ( 88 sq mi ) Urban 285 km ( 110 sq mi ) Elevation 55.8 -- 91.5 m ( 183.1 -- 300.2 ft ) Population ( 2011 ) Capital city 1,883,425 Estimate ( 2016 ) 2,106,144 Rank 1st in Romania ( 6th in Bucharest destroying a third of the city . Ottoman massacre of Greek irregulars in Bucharest ( August , 1821 ) In 1862 , after Wallachia and Moldavia were united to form the Principality of Romania , Bucharest became the new nation 's capital city . In 1881 , it became the political centre of the newly proclaimed Kingdom of Romania under King Carol I. During the second half of the 19th century , the city 's population increased dramatically , and a new period of urban development began . During this period , gas lighting , horse - drawn trams , and limited electrification were introduced . The Dâmbovița River was also massively channelled in 1883 , thus putting a stop to previously endemic floods like the 1865 flooding of Bucharest . The Fortifications of Bucharest were built . The extravagant architecture and cosmopolitan high culture of this period won Bucharest the nickname of `` Little Paris '' ( Micul Paris ) of the east , with Calea Victoriei as its Champs - Élysées . I.C. Brătianu Boulevard in the 1930s Between 6 December 1916 and November 1918 , the city was occupied by German forces as a result of the Battle of Bucharest , with the official capital temporarily moved to Iași , in the Moldavia region . After World War I , Bucharest became the capital of Greater Romania . In the interwar years , Bucharest 's urban development continued , with the city gaining an average of 30,000 Bucharest documents in 1459 . It became the capital of Romania in 1862 and is the centre of Romanian media , culture , and art . Its architecture is a mix of historical ( neo-classical ) , interbellum ( Bauhaus and art deco ) , communist - era and modern . In the period between the two World Wars , the city 's elegant architecture and the sophistication of its elite earned Bucharest the nickname of `` Little Paris '' ( Micul Paris ) . Although buildings and districts in the historic city centre were heavily damaged or destroyed by war , earthquakes , and above all Nicolae Ceaușescu 's program of systematization , many survived . In recent years , the city has been experiencing an economic and cultural boom . In 2016 , the historical city centre was listed as `` endangered '' by the World Monuments Watch . According to the 2011 census , 1,883,425 inhabitants live within the city limits , a decrease from the 2002 census . Adding the satellite towns around the urban area , the proposed metropolitan area of Bucharest would have a population of 2.27 million people . According to Eurostat , Bucharest has a functional urban area of 2,412,530 residents ( as of 2015 ) . Bucharest is the sixth - largest city in the European Union by population within city limits , after London , Berlin , Madrid , Rome , and Paris . Economically , Bucharest is the most prosperous
Copyright © 2023 IBM. This notebook and its source code are released under the terms of the MIT License.